WO2024252494A1 - Video processing device, method, and program - Google Patents

Abstract

A video processing device according to one embodiment of the present invention comprises: a video feature extraction unit that extracts feature information indicating a feature of a video; and an estimation unit that estimates, on the basis of a feature based on a relationship between pieces of language information related to an action of a subject appearing in the video and the feature information extracted by the video feature extraction unit, a composite action that is a combination of basic actions indicating a general action of the subject.

Description

Image processing device, method and program

Embodiments of the present invention relate to a video processing device, method, and program.

Technology that recognizes the behavior of people in video footage is expected to be useful for work analysis by automatically recognizing and recording tasks that involve human physical movements, such as work in retail stores or medical settings, and much research is being conducted into this technology.
For example, the method disclosed in Non-Patent Document 1 prepares a video dataset annotated with the action labels of the recognition target as training data in advance, and acquires a model trained to recognize actions shown in the input video. In the action recognition technology disclosed in Non-Patent Document 1, the action labels defined in the video dataset are the recognition target, and these action labels are basic action labels as generic action labels such as actions that may commonly occur in the business of various industries, for example, "walking", "holding" or "putting".

Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, Christoph Feichtenhofer, “MViTv2: Improved Multiscale Vision Transformers for Classification and Detection.” In proc. of CVPR 2022.

Here, an action label specific to a certain industry, for example, "displaying a product," can be called a composite action label, which is a combination of the basic action labels "holding a product" and "placing a product." In order to estimate the above composite action label using the method disclosed in the above non-patent document 1, it is necessary to prepare a video dataset annotated with composite action labels that may occur for each industry, which entails a great deal of cost.

This invention was made in light of the above circumstances, and its purpose is to provide a video processing device, method, and program that can appropriately estimate the combination of units of the actions of a person shown in a video.

An image processing device according to one embodiment of the present invention includes an image feature extraction unit that extracts feature information indicative of features of an image, and an estimation unit that estimates a composite behavior, which is a combination of basic behaviors indicative of a general-purpose behavior of a subject, based on features based on the relationship between linguistic information related to the behavior of a subject appearing in the image and the feature information extracted by the image feature extraction unit.

The image processing method according to one aspect of the present invention is a method performed by an image processing device, and includes: extracting feature information indicating features of an image by an image feature extraction unit of the image processing device; and estimating, by an estimation unit of the image processing device, a composite behavior that is a combination of basic behaviors indicating general-purpose behavior of the subject, based on features based on the relationship of linguistic information related to the behavior of the subject appearing in the image and the feature information extracted by the image feature extraction unit.

The present invention makes it possible to appropriately estimate the combination of units of a person's actions shown in a video.

FIG. 1 is a diagram showing an application example of a video processing device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of the concept of each function of the video processing device according to one embodiment of the present invention. FIG. 3 is a diagram showing an example of an action dictionary in the medical and nursing industry. Figure 4 is a flow chart showing a first example of a learning processing routine executed by the video processing device. FIG. 5 is a flowchart showing a second example of a learning process routine executed by the video processing device. FIG. 6 is a flowchart showing a first example of an inference processing routine executed by the video processing device. FIG. 7 is a flowchart showing a second example of an inference processing routine executed by the video processing device. Figure 8 is a block diagram showing an example of the hardware configuration of an image processing device according to one embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
<Configuration>
First, a configuration of a video processing device according to an embodiment of the present invention will be described. Fig. 1 is a diagram showing an application example of a video processing device according to an embodiment of the present invention.
The video processing device 100 shown in FIG. 1 is configured as a computer having a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) in which a program for executing a video processing routine described below is stored, and is functionally configured as follows.

As shown in FIG. 1, the video processing device 100 according to this embodiment includes a memory unit 1, a video feature extraction unit 2, a label feature creation unit 3, a basic behavior inference unit 4, a composite behavior inference unit 5, a parameter update unit 6, an output unit 7, and a correct composite behavior label creation unit 8.

The storage unit 1 stores a video x, an action dictionary D, a correct basic action label j, a pseudo correct composite action label l, a video feature extraction model a, a label feature creation model b, a basic action inference model c, a composite action inference model d, and a correct composite action label estimator e.

FIG. 2 is a diagram showing an example of the concept of each function of the video processing device according to one embodiment of the present invention.
In this Figure 2, an example of a concept showing the configuration of an image x, an action dictionary D, a correct basic action label j, a pseudo correct composite action label l, an image feature extraction model a, a label feature creation model b, a basic action inference model c, a composite action inference model d, and a correct composite action label estimator e is shown.

Image x may be any data that constitutes certain video data, for example a collection of images of multiple frames. In addition, image x may be provided with depth information for each frame and optical flow information that indicates changes in appearance between frames.

The behavior dictionary D is a dictionary in which nodes are linguistic information that represents a person's behavior, edges are relational labels y that indicate the relationship between the linguistic information of each node, and the relationships between the basic behavior labels and composite behavior labels below, as well as the relationships between the composite behavior labels, are structured. The data in the behavior dictionary D can be any data that is composed of these nodes and edges. Figure 3 shows an example of the behavior dictionary D in the medical and nursing industry.

In the example shown in Figure 3, "holding a thermometer," "placing a thermometer," and "holding a compress" are basic action labels in the medical and nursing industry. Also, in the example shown in Figure 3, "taking body temperature" is one of the composite action labels that combines the basic action labels "holding a thermometer" and "placing a thermometer."

The correct basic action label j can be any label that indicates the basic action of the person appearing in the video x, i.e., any correct information about linguistic information related to the basic action.

Here, basic actions are generic actions that can occur in a variety of industries. For example, "walking," "touching," or "holding." When video x is a collection of images with N frames, basic actions are a set of N labels that indicate the basic actions of people in each frame.

The pseudo-correct composite action label l is a label indicating the composite action of a person appearing in the video x, i.e., correct information of linguistic information related to basic actions, and can be any correct information of the composite action label output by the correct composite action label estimator e by inputting the action dictionary D and the correct basic action label j.

Here, a composite action label is a label that indicates an action that may occur in a particular industry, for example, "take temperature" or "cool down" in the medical and nursing industry. When video x is a set of images with N frames, a composite action label is M composite action labels output from action dictionary D, which is obtained by inputting a set of N labels that indicate the basic actions of people in each frame into action dictionary D.

The image feature extraction model a may be any neural network that inputs an image x and outputs a tensor indicating the features of the image x. For example, the image feature extraction model a is MViT (Multiscale Vision Transformers) disclosed in Non-Patent Document 1.
The image feature x′ may be any tensor that indicates the features of the image x, and is, for example, the output tensor when the image x is input to MViT disclosed in Non-Patent Document 1.

The label feature creation model b can be any neural network that inputs a relational label y indicating the relationship between each node in the behavior dictionary D and outputs a vector indicating the features of the relational label y, for example a neural network with one fully connected layer.

The relational label feature y' can be any vector that indicates the features of the relational label y, for example the output tensor when the relational label y is input to a neural network with one fully connected layer.

The basic behavior inference model c may be any neural network that inputs the video feature x' and outputs a basic behavior label.
The composite behavior inference model d may be any neural network that receives an image feature x' and a relational label feature y' and outputs a composite behavior label.

The correct composite behavior label estimator e may be any system that has the function of inputting the behavior dictionary D and the correct basic behavior label j and outputting the pseudo correct composite behavior label l. For example, the correct composite behavior label estimator e may be a program that searches the behavior dictionary D for a node that contains the correct basic behavior label j, and outputs multiple nodes linked to that node by the relationship label y as a composite behavior label.

The image feature extraction unit 2 receives the image x and the image feature extraction model a from the storage unit 1, inputs the image x to the image feature extraction model a, and outputs the image feature x'.
The label feature creation unit 3 receives the relational label y and the label feature creation model b from the storage unit 1, inputs the relational label y to this label feature creation model b, and outputs the relational label feature y'.

The basic behavior inference unit 4 receives the basic behavior inference model c from the memory unit 1 and receives the video feature x' from the video feature extraction unit 2, inputs this video feature x' to the basic behavior inference model c, and outputs the estimated basic behavior label i.

The composite behavior inference unit 5 receives the composite behavior inference model d from the memory unit 1, receives the video feature x' from the video feature extraction unit 2, and receives the relational label feature y' from the label feature creation unit 3, inputs the video feature x' and the relational label feature y' to the composite behavior inference model d, and outputs the estimated composite behavior label k.

The parameter update unit 6 receives the video feature extraction model a, the label feature creation model b, the basic behavior inference model c, the composite behavior inference model d, the correct basic behavior label j, and the pseudo-correct composite behavior label l from the memory unit 1, receives the estimated basic behavior label i from the basic behavior inference unit 4, and receives the estimated composite behavior label k from the composite behavior inference unit 5, and updates the parameters of each neural network so as to satisfy the following two constraints.

The first constraint is to update the parameters of each neural network so that the estimated basic action label i and the correct basic action label j approach or become the same, and any learning method that is set to satisfy this constraint is acceptable. The difference between the estimated basic action label i and the correct basic action label j is also called loss.

In this case, each neural network may be a video feature extraction model a, a label feature creation model b, a basic behavior inference model c, and a composite behavior inference model d, or a video feature extraction model a, a label feature creation model b, and a basic behavior inference model c, or a label feature creation model b and a basic behavior inference model c, or a video feature extraction model a and a basic behavior inference model c, or just a basic behavior inference model c.

For example, the parameter update unit 6 calculates the cross-entropy error between the estimated basic behavior label i and the correct basic behavior label j as shown in the following formula (1), and updates the parameters of the video feature extraction model a, the label feature creation model b, the basic behavior inference model c, and the composite behavior inference model d so as to reduce this error or set it to zero.

Here, m in formula (1) is an index of the estimated basic behavior label i and the correct basic behavior label j, _ym is a value in the estimated basic behavior label i, and _tm is a value in the correct basic behavior label j. In this _tm , only the index value of the correct class is "1", and the other index values are "0".

The second constraint is to update the parameters of each neural network so that the estimated composite action label k and the pseudo-ground truth composite action label l approach or become the same, and any learning method that is configured to satisfy this constraint is acceptable.

In this case, each neural network may be composed of a video feature extraction model a, a label feature creation model b, a basic behavior inference model c, and a composite behavior inference model d, or may be composed of a video feature extraction model a, a label feature creation model b, and a composite behavior inference model d, or may be composed of a label feature creation model b and a composite behavior inference model d, or may be composed of a video feature extraction model a and a composite behavior inference model d, or may be composed of only a composite behavior inference model d.

For example, the parameter update unit 6 calculates the cross-entropy error for the estimated composite behavior label k and the pseudo-ground-truth composite behavior label l as in the above formula (1), and updates the parameters of the video feature extraction model a, the label feature creation model b, the basic behavior inference model c, and the composite behavior inference model d so as to reduce this error or make it zero.

Here, m in formula (1) is an index of the estimated composite behavior label k and the pseudo correct composite behavior label l, _ym is a value in the estimated composite behavior label k, and _tm is a value in the pseudo correct composite behavior label l. In this _tm, only the index value of the correct class is "1", and the other index values are "0".

The output unit 7 receives the estimated basic behavior label i from the basic behavior inference unit 4 and receives the estimated composite behavior label k from the composite behavior inference unit 5, and outputs the inference result.
The estimation result may be only the estimated composite behavior label k, or may include the estimated basic behavior label i in addition to the estimated composite behavior label k.

The correct composite behavior label creation unit 8 receives the correct composite behavior label estimator e, the behavior dictionary D, and the correct basic behavior label j from the memory unit 1, inputs the behavior dictionary D and the correct basic behavior label j to the correct composite behavior label estimator e, and outputs the pseudo-correct composite behavior label l.

<Function of the image processing device>
Next, the operation of the image processing device 100 according to this embodiment will be described.
The video processing device 100 executes a learning processing routine and an inference processing routine, which will be described below.

<<Learning processing routine>>
First, a first example of the learning process routine will be described with reference to Fig. 4. The first example of the learning process routine executed by the video processing device is shown in a flowchart in Fig. 4.
In step S101, the correct composite behavior label creation unit 8 receives the correct composite behavior label estimator e, the behavior dictionary D, and the correct basic behavior label j from the storage unit 1. The correct composite behavior label creation unit 8 inputs the behavior dictionary D and the correct basic behavior label j to the correct composite behavior label estimator e, and transmits the pseudo correct composite behavior label l output from the correct composite behavior label estimator e to the storage unit 1.

In step S102, the video feature extraction unit 2 receives the video x and the video feature extraction model a from the memory unit 1. The video feature extraction unit 2 inputs the video x to this video feature extraction model a, and transmits the video feature x' output from this video feature extraction model a to the basic behavior inference unit 4 and the composite behavior inference unit 5.

In step S103, the label feature creation unit 3 receives the relational label y and the label feature creation model b from the memory unit 1, inputs the relational label y to the label feature creation model b, and transmits the relational label feature y' output from the label feature creation model b to the composite behavior inference unit 5.

In step S104, the basic behavior inference unit 4 receives the basic behavior inference model c from the memory unit 1, and receives the video feature x' from the video feature extraction unit 2. The basic behavior inference unit 4 inputs this video feature x' to the basic behavior inference model c, and transmits the estimated basic behavior label i output from this basic behavior inference model c to the parameter update unit 6.

In step S105, the composite behavior inference unit 5 receives the composite behavior inference model d from the memory unit 1, receives the video feature x' from the video feature extraction unit 2, and receives the relational label feature y' from the label feature creation unit 3. The composite behavior inference unit 5 inputs the video feature x' and the relational label feature y' to the composite behavior inference model d, and transmits the estimated composite behavior label k output from this composite behavior inference model d to the parameter update unit 6.

In step S106, the parameter update unit 6 receives the video feature extraction model a, the label feature creation model b, the basic behavior inference model c, the composite behavior inference model d, the correct basic behavior label j, and the pseudo-correct composite behavior label l from the memory unit 1, receives the estimated basic behavior label i from the basic behavior inference unit 4, and receives the estimated composite behavior label k from the composite behavior inference unit 5. The parameter update unit 6 updates the parameters of each neural network so as to satisfy the above two constraints.

Next, a second example of the learning process routine will be described below with reference to a flowchart of FIG 5, which shows the second example of the learning process routine executed by the video processing device.
First, the processes of steps S101, S102, S103 and S105 described in the first example of the learning process routine are carried out in that order.

Next, in step S107, the parameter update unit 6 receives the video feature extraction model a, the label feature creation model b, the composite behavior inference model d, and the pseudo-ground-truth composite behavior label l from the memory unit 1, receives the estimated composite behavior label k from the composite behavior inference unit 5, and updates the parameters of each neural network to satisfy the constraint that the estimated composite behavior label k and the pseudo-ground-truth composite behavior label l approach or become the same.

<<Inference processing routine>>
Next, a first example of an inference processing routine will be described below with reference to a flowchart shown in FIG 6. The first example of an inference processing routine executed by the video processing device is shown in FIG.
First, the processes of steps S102, S103, S104 and S105 described in the first example of the learning process routine are carried out in that order.

Next, in step S206, the output unit 7 receives the estimated basic behavior label i from the basic behavior inference unit 4 and receives the estimated composite behavior label k from the composite behavior inference unit 5, and outputs the received estimated composite behavior label k and estimated basic behavior label i.

Next, a second example of the inference processing routine will be described with reference to Fig. 7. The second example of the inference processing routine executed by the video processing device is shown in a flowchart in Fig. 7.
First, steps S102, S103 and S105 described in the second example of the learning process routine are carried out in this order.
Next, in step S207, the output unit 7 receives the estimated composite action label k from the composite action inference unit 5, and outputs the received estimated composite action label k.

According to the video processing device of one embodiment of the present invention described above, an action dictionary is constructed that structures the relationships between basic action labels and composite action labels and the relationships between the composite action labels, and pseudo-correct composite action labels are estimated from correct basic action labels, thereby creating learning data annotated with correct basic action labels and pseudo-correct composite action labels, and a model is learned that infers composite action labels simultaneously with basic action labels by utilizing the created learning data.
This has the effect of making it possible to estimate industry-specific composite action labels while eliminating the cost of building industry-specific video datasets.

FIG. 8 is a block diagram showing an example of a hardware configuration of a video processing device according to an embodiment of the present invention.
8, the video processing device 100 according to the embodiment is configured, for example, by a server computer or a personal computer, and has a hardware processor 111A such as a CPU. A program memory 111B, a data memory 112, an input/output interface 113, and a communication interface 114 are connected to the hardware processor 111A via a bus 115.

The communication interface 114 includes, for example, one or more wireless communication interface units, and enables the transmission and reception of information to and from the communication network NW. As a wireless interface, for example, an interface that adopts a low-power wireless data communication standard such as a wireless LAN (Local Area Network) is used.

The input/output interface 113 is connected to an input device 200 and an output device 300 that are attached to the video processing device 100 and are used by users, etc.

The input/output interface 113 takes in operation data input by a user or the like through an input device 200 such as a keyboard, touch panel, touchpad, mouse, etc., and outputs the output data to an output device 300 including a display device using liquid crystal or organic EL (Electro Luminescence), etc., for display. The input device 200 and the output device 300 may be devices built into the video processing device 100, or may be input devices and output devices of other computers or information terminals that can communicate with the video processing device 100 via the network NW.

The program memory 111B is a non-transient tangible storage medium that combines a non-volatile memory such as a hard disk drive (HDD) or solid state drive (SSD) that can be written to and read from at any time, with a non-volatile memory such as a ROM, and stores the programs necessary to execute various control processes, etc. according to one embodiment.

The data memory 112 is a tangible storage medium that is, for example, a combination of the non-volatile memory described above and a volatile memory such as RAM, and is used to store various data acquired and created during various processing steps.

The video processing device 100 according to one embodiment of the present invention can be configured as an information processing device having a software-based processing function unit. This processing function unit corresponds to each unit of the video processing device 100 in FIG. 1.

The storage device used as a work memory or the like by the video processing device 100 can be configured by using the data memory 112 shown in FIG. 11. However, these configured storage areas are not essential components within the video processing device 100, and may be areas provided in a storage device such as an external storage medium such as a USB (Universal Serial Bus) memory, or a database server located in the cloud.

Any of the processing function units in the above-mentioned components can be realized by having the hardware processor 111A read and execute a program stored in the program memory 111B. Note that some or all of these processing function units may be realized in a variety of other forms, including integrated circuits such as application specific integrated circuits (ASICs (Application Specific Integrated Circuits)) or field-programmable gate arrays (FPGAs).

The methods described in each embodiment can be stored as a program (software means) that can be executed by a computer on a recording medium such as a magnetic disk (floppy disk, hard disk, etc.), optical disk (CD-ROM, DVD, MO, etc.), semiconductor memory (ROM, RAM, flash memory, etc.), and can be distributed by transmitting it via a communication medium. The programs stored on the medium also include a setting program that configures the software means (including not only execution programs but also tables and data structures) that the computer executes. The computer that realizes this device reads the program recorded on the recording medium, and in some cases configures the software means using the setting program, and executes the above-mentioned processing by having the operation controlled by this software means. The recording medium referred to in this specification is not limited to a recording medium for distribution, but also includes storage media such as a magnetic disk or semiconductor memory installed inside the computer or in a device connected via a network.

The present invention is not limited to the above-described embodiments, and can be modified in various ways during implementation without departing from the gist of the invention. The embodiments may also be implemented in appropriate combination, in which case the combined effects can be obtained. Furthermore, the above-described embodiments include various inventions, and various inventions can be extracted by combinations selected from the multiple constituent elements disclosed. For example, if the problem can be solved and an effect can be obtained even if some constituent elements are deleted from all the constituent elements shown in the embodiments, the configuration from which these constituent elements are deleted can be extracted as an invention.

REFERENCE SIGNS LIST 100: video processing device 1: memory unit 2: video feature extraction unit 3: label feature creation unit 4: basic behavior inference unit 5: composite behavior inference unit 6: parameter update unit 7: output unit 8: correct composite behavior label creation unit

Claims (7)

a video feature extraction unit that extracts feature information indicating features of the video;
an estimation unit that estimates a composite behavior, which is a combination of basic behaviors that indicate a general-purpose behavior of the subject, based on features based on a relationship between linguistic information related to the behavior of the subject captured in the video and the feature information extracted by the video feature extraction unit;
A video processing device comprising: The estimation unit is
Further estimating the basic behavior of the subject appearing in the video based on the feature information extracted by the video feature extraction unit.
The video processing device according to claim 1 . The estimation unit is
inputting features based on a relationship between linguistic information related to the subject's behavior and the feature information extracted by the video feature extraction unit into a neural network, and estimating the composite behavior based on the results of this input;
and an update unit that updates parameters of the neural network so that the composite behavior estimated using the neural network approaches correct information of the composite behavior.
The video processing device according to claim 1 . The estimation unit is
inputting the feature information extracted by the video feature extraction unit into a neural network, and estimating the basic behavior based on the result of this input;
An update unit that updates parameters of the neural network so that the basic behavior estimated using the neural network approaches correct answer information of the basic behavior.
The video processing device according to claim 2 . The estimation unit is
estimating correct answer information for the composite behavior based on an behavior dictionary in which relationships between basic behaviors and composite information of the subject and relationships between a plurality of types of composite behaviors are structured, and correct answer information for the basic behaviors;
The image processing device according to claim 3 . A method performed by a video processing device, comprising:
extracting feature information indicating features of the video by a video feature extraction unit of the video processing device;
an estimation unit of the video processing device estimates a composite behavior, which is a combination of basic behaviors that indicate general-purpose behavior of the subject, based on features based on a relationship between linguistic information related to the behavior of the subject captured in the video and the feature information extracted by the video feature extraction unit;
A video processing method comprising: An image processing program that causes a processor to function as each part of the image processing device according to any one of claims 1 to 5.

Images