JP7500246B2 - Imaging device, control method, and program - Google Patents

Description

The present invention relates to technology for improving processing efficiency.

In recent years, in various scenes, images captured by surveillance cameras are used for image analysis, such as detecting and tracking objects and estimating attributes, and estimating the number of objects based on the results of such image analysis. Conventionally, such image processing has been performed by transferring surveillance camera footage to a high-performance computing device such as a PC or server that performs the actual image processing. In contrast, with the recent improvement in the processing power of mobile computing devices, it has become possible to perform image processing on the surveillance camera side. The processing on the camera side can be performed, for example, by a computing device arranged in the camera body. Also, by arranging the computing device in a removable device such as a USB, the removable device can perform at least part of the processing.

International Publication No. 2013/011988

In devices that can accommodate removable devices, it is essential to provide control that improves convenience depending on whether or not the removable device is attached.

The present invention provides a technology that improves the efficiency of having a removable device execute processing in an imaging device to which the removable device is attached.

An imaging device according to one aspect of the present invention is an imaging device having a mounting part to which a device having a function of performing image analysis processing on a captured image can be detachably attached, and in order to calculate a processing time, which is the time from a predetermined timing associated with the start of the image analysis processing by the device attached to the mounting part to the completion of the processing , the imaging device has an acquisition means for acquiring information from the device including the number of clock cycles from the start of the image analysis processing to its completion, calculating the processing time based on the information, and , when the device is caused to start executing the image analysis processing, acquiring the results of the image analysis processing from the device after the processing time has elapsed from the predetermined timing associated with the start of the image analysis processing.

According to the present invention, in an imaging device to which a removable device is attached, it is possible to improve the efficiency of having the removable device execute processing.

FIG. 1 illustrates an example of a system configuration. FIG. 2 is a diagram illustrating an example of a hardware configuration of an imaging apparatus. FIG. 2 is a diagram illustrating an example of a functional configuration of an imaging apparatus. FIG. 2 illustrates an example of the hardware configuration of a removable device. FIG. 1 illustrates an example of a functional configuration of a removable device. FIG. 2 is a diagram illustrating an example of a flow of processing executed in the system. FIG. 13 is a diagram illustrating an example of a process flow for grasping an analysis process. FIG. 11 is a diagram illustrating an example of a process flow for determining the content of an analysis process. FIG. 13 is a diagram illustrating an example of a control flow for executing an analysis process. FIG. 13 is a diagram illustrating an example of a control flow for executing post-processing. FIG. 13 is a diagram illustrating an example of a hardware configuration of a removable device according to a modified example. FIG. 2 is a diagram showing command and response structures. FIG. 13 is a diagram illustrating an example of the flow of a result acquisition process taking into consideration a case where the process is incomplete.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

<System Configuration>
FIG. 1 shows an example of the configuration of an image analysis system according to the present embodiment. In the following, as an example, a case where this system is a specific person tracking system will be described. However, the present system is not limited to this, and the following discussion can be applied to any system that analyzes images and outputs predetermined information. This system is configured to include imaging devices 110a to 110d, a network 120, and an input/output device 130. Note that the imaging devices 110a to 110d each have a slot into which a device capable of recording captured images can be attached and detached, and are connected to the removable devices 100a to 100d by inserting the removable devices 100a to 100d into the slot. Note that, in the following, the removable devices 100a to 100d are referred to as "detachable devices 100", and the imaging devices 110a to 110d are referred to as "imaging devices 110".

The removable device 100 is a computing device that is removable from the imaging device 110. As an example, the removable device 100 is a device in which a predetermined processing circuit is mounted on an SD card. The removable device 100 is configured to be able to be inserted into the imaging device 110 in its entirety, for example, in the form of an SD card, so that it can be connected to the imaging device 110 with no part protruding from the imaging device 110. The removable device 100 may also be configured to be able to be inserted into the imaging device 110 with more than half of it being inserted into the imaging device 110, so that it can be connected to the imaging device 110 with only a small part protruding from the imaging device 110. This makes it possible to prevent the removable device 100 from interfering with obstacles such as wiring, and to increase the convenience of using the device. In addition, since many existing imaging devices 110 such as network cameras are provided with an SD card slot, the removable device 100 can provide an extended function to the existing imaging device 110. Note that the removable device 100 may be configured to be attached to the imaging device 110 using any interface other than the SD card that is used when a storage device capable of storing at least images captured by the imaging device 110 is attached. For example, the removable device 100 may have a USB (Universal Serial Bus) interface and be configured to be attached to a USB socket of the imaging device 110. In addition, the specified processing circuit is implemented, for example, by an FPGA (Field Programmable Gate Array) programmed to execute a specified process, but may be implemented in other formats.

The imaging device 110 is an imaging device such as a network camera. In this embodiment, the imaging device 110 is assumed to have a built-in arithmetic device capable of processing images, but is not limited to this. For example, an external computer such as a PC (personal computer) connected to the imaging device 110 may exist, and the combination of these may be treated as the imaging device 110. In addition, in this embodiment, it is assumed that the removable device 100 is attached to all of the imaging devices 110. Note that, in FIG. 1, four imaging devices 110 and removable devices attached to each of them are shown, but the number of combinations of these devices may be three or less, or five or more. By attaching the removable device 100 having an image analysis processing function to the imaging device 110, it becomes possible to perform image processing on the imaging device 110 side even if the imaging device 110 does not have an image analysis processing function. Furthermore, in a configuration in which a computing device for image processing is arranged in the imaging device 110 as in this embodiment, the removable device 100 in which the computing device is arranged is attached to the imaging device 110, thereby making it possible to diversify and advance the image processing that can be performed on the imaging device 110 side.

The input/output device 130 is a device that receives input from a user and outputs information to the user (e.g., displays information). In this embodiment, the input/output device 130 is, for example, a computer such as a PC, and inputs and outputs information by executing a browser or native application installed on the computer by a built-in processor.

The imaging device 110 and the input/output device 130 are connected to each other via a network 120 so that they can communicate with each other. The network 120 includes a plurality of routers, switches, cables, etc. that meet a communication standard such as Ethernet (registered trademark). In this embodiment, the network 120 may be any network that enables communication between the imaging device 110 and the input/output device 130, and may be constructed with any scale, configuration, and communication standard that it complies with. For example, the network 120 may be the Internet, a wired LAN (Local Area Network), a wireless LAN, a WAN (Wide Area Network), etc. In addition, the network 120 may be configured to enable communication using a communication protocol that complies with the ONVIF (Open Network Video Interface Forum) standard. However, the present invention is not limited to this, and the network 120 may be configured to enable communication using other communication protocols such as a proprietary communication protocol.

<Device Configuration>
(Configuration of the imaging device)
Next, a description will be given of the configuration of the imaging device 110. Fig. 2 is a diagram showing an example of the hardware configuration of the imaging device 110. The imaging device 110 includes, as its hardware configuration, for example, an imaging unit 201, an image processing unit 202, an arithmetic processing unit 203, a distribution unit 204, and an SD I/F unit 205. Note that I/F is an abbreviation for interface.

The imaging unit 201 includes a lens unit for forming an image of light, and an imaging element that converts the formed light into an analog signal. The lens unit has a zoom function for adjusting the angle of view, and an aperture function for adjusting the amount of light. The imaging element has a gain function for adjusting the sensitivity when converting light into an analog signal. These functions are adjusted based on setting values notified by the image processing unit 202. The analog signal acquired by the imaging unit 201 is converted into a digital signal by an analog-digital conversion circuit, and transferred to the image processing unit 202 as an image signal.

The image processing unit 202 is configured to include an image processing engine and its peripheral devices. The peripheral devices include, for example, a RAM (Random Access Memory) and drivers for each I/F. The image processing unit 202 performs image processing, such as development processing, filtering, sensor correction, and noise removal, on the image signal acquired from the imaging unit 201 to generate image data. The image processing unit 202 can also transmit setting values to the lens unit and the imaging element, and perform exposure adjustment so that an appropriately exposed image can be acquired. The image data generated in the image processing unit 202 is transferred to the calculation processing unit 203.

The arithmetic processing unit 203 is composed of one or more processors such as a CPU or MPU, memories such as a RAM or ROM, drivers for each I/F, etc. Note that CPU is an acronym for Central Processing Unit, MPU is an acronym for Micro Processing Unit, RAM is an acronym for Random Access Memory, and ROM is an acronym for Read Only Memory. In one example, the arithmetic processing unit 203 determines whether each part of the processing to be executed in the above-mentioned system will be executed by the imaging device 110 or the removable device 100, and can execute the processing corresponding to the determined allocation. The details of this processing content and the allocation of processing will be described later. The image received from the image processing unit 202 is transferred to the distribution unit 204 or the SD I/F unit 205. In addition, data on the processing result and processing status is also transferred to the distribution unit 204.

The distribution unit 204 includes a network distribution engine and peripheral devices such as a RAM and an ETH PHY module. The ETH PHY module is a module that executes processing of the physical (PHY) layer of Ethernet. The distribution unit 204 converts the image data, processing results, and processing status data acquired from the arithmetic processing unit 203 into a format that can be distributed to the network 120, and outputs the converted data to the network 120. The SD I/F unit 205 is an interface part for connecting to the removable device 100, and includes, for example, a power source and an attachment mechanism such as a removable socket for attaching and detaching the removable device 100. Here, it is assumed that the SD I/F unit 205 is configured in accordance with the SD standard established by the SD Association. Communications between the removable device 100 and the imaging device 110, such as transferring images acquired from the calculation processing unit 203 to the removable device 100 and acquiring data from the removable device 100, are performed via the SD I/F unit 205.

Figure 3 shows an example of the functional configuration of the imaging device 110. The imaging device 110 includes, as its functions, an imaging control unit 301, a signal processing unit 302, a storage unit 303, a control unit 304, an analysis unit 305, a device communication unit 306, and a network communication unit 307, for example.

The imaging control unit 301 executes control to capture the surrounding environment via the imaging unit 201. The signal processing unit 302 performs a predetermined process on the image captured by the imaging control unit 301 to generate captured image data. Hereinafter, this captured image data will simply be referred to as the "captured image". For example, the signal processing unit 302 encodes the image captured by the imaging control unit 301. For still images, the signal processing unit 302 encodes using an encoding method such as JPEG (Joint Photographic Experts Group). For moving images, the signal processing unit 302 encodes using an encoding method such as H.264/MPEG-4 AVC (hereinafter referred to as "H.264") and HEVC (High Efficiency Video Coding). Furthermore, the signal processing unit 302 may encode the image using an encoding method selected by the user from among a plurality of encoding methods set in advance, for example, via an operation unit (not shown) of the imaging device 110.

The memory unit 303 stores a list of analysis processes that can be executed by the analysis unit 305 (hereinafter referred to as the "first processing list") and a list of post-processing for the results of the analysis processes. The memory unit 303 also stores the results of the analysis processes described below. Note that in this embodiment, the process executed is an analysis process, but any process may be executed, and the memory unit 303 may store a first processing list and a post-processing list for processes related to the process executed. The control unit 304 controls the signal processing unit 302, the memory unit 303, the analysis unit 305, the device communication unit 306, and the network communication unit 307 so that each executes a predetermined process.

The analysis unit 305 selectively performs at least one of pre-analysis processing, analysis processing, and post-analysis processing, which will be described later, on the captured image. The pre-analysis processing is processing performed on the captured image before performing the analysis processing, which will be described later. In the pre-analysis processing of this embodiment, as an example, a process of dividing the captured image to create divided images is performed. The analysis processing is processing of outputting information obtained by analyzing an input image. In the analysis processing of this embodiment, as an example, a process of performing at least one of human body detection processing, face detection processing, and vehicle detection processing using the divided image obtained by the pre-analysis processing as input, and outputting the analysis processing result is performed. The analysis processing may be processing configured to output the position of an object in a divided image using a machine learning model that has been trained to detect an object included in an image, for example, by the technology of Non-Patent Document 1. The post-analysis processing is processing performed after the analysis processing is performed. In the post-analysis processing of this embodiment, as an example, a process of outputting the total number of objects detected in each divided image based on the analysis processing result for each divided image is performed. The analysis process may involve pattern matching to detect objects in an image and output their positions.

The device communication unit 306 communicates with the removable device 100. The device communication unit 306 converts input data into a format that can be processed by the removable device 100, and transmits the data obtained by the conversion to the removable device 100. The device communication unit 306 also receives data from the removable device 100 and converts the received data into a format that can be processed by the imaging device 110. In this embodiment, the device communication unit 306 executes a process of converting decimals between a floating-point format and a fixed-point format as a conversion process, but this is not limited to this, and other processes may be executed by the device communication unit 306. In this embodiment, the device communication unit 306 communicates with the removable device 100 by transmitting a command sequence that is predetermined within the range of the SD standard to the removable device 100 and receiving a response from the removable device 100. The network communication unit 307 communicates with the input/output device 130 via the network 120.

(Configuration of removable devices)
4 shows an example of the hardware configuration of the removable device 100. The removable device 100 includes, as an example, an I/F unit 401, an FPGA 402, an SD controller 403, and a storage unit 404. The removable device 100 is formed in a shape that can be inserted into and removed from a removable socket of the SD I/F unit 205 of the imaging device 110, that is, in a shape that complies with the SD standard.

The I/F unit 401 is an interface section for connecting devices such as the imaging device 110 to the removable device 100. The I/F unit 401 is configured to include electrical contact terminals and the like that receive power from the imaging device 110 and generate and distribute power to be used within the removable device 100. As with the SD I/F unit 205 of the imaging device 110, the I/F unit 401 complies with items defined (compliant) within the SD standard. Receiving images and setting data from the imaging device 110 and transmitting data from the FPGA 402 to the imaging device 110 are performed via the I/F unit 401.

The FPGA 402 includes an input/output control unit 410, a processing switching unit 411, and an arithmetic processing unit 412. The FPGA 402 is a type of semiconductor device that can repeatedly reconfigure the internal logic circuit structure. The processing realized by the FPGA 402 can add (provide) a processing function to the device to which the removable device 100 is attached. In addition, the reconfiguration function of the FPGA 402 allows the logic circuit structure to be changed later, so that, for example, by attaching the removable device 100 to a device in a field where technology is rapidly advancing, it becomes possible to execute appropriate processing at the appropriate time in the device. In this embodiment, an example in which an FPGA is used will be described, but as long as the processing described later can be realized, for example, a general-purpose ASIC or a dedicated LSI may be used. The FPGA 402 is started by writing setting data including information on the logic circuit structure to be generated from a dedicated I/F, or by reading the setting data from the dedicated I/F. In this embodiment, it is assumed that this setting data is stored in the storage unit 404. When the FPGA 402 is powered on, it reads the setting data from the storage unit 404, generates a logic circuit, and starts it up. However, this is not limited to the above, and for example, the imaging device 110 may write the setting data to the FPGA 402 via the I/F unit 401 by implementing a dedicated circuit in a removable device.

The input/output control unit 410 includes a circuit for transmitting and receiving images to and from the imaging device 110, a circuit for analyzing commands received from the imaging device 110, and a circuit for controlling based on the analysis results. The commands here are defined in the SD standard, and the input/output control unit 410 can detect some of them. Details of the functions will be described later. The input/output control unit 410 controls to transmit an image to the SD controller 403 in the case of storage processing, and to transmit an image to the arithmetic processing unit 412 in the case of image analysis processing. In addition, when the input/output control unit 410 receives setting data for switching processing, it transmits the setting data to the processing switching unit 411. The processing switching unit 411 includes a circuit for acquiring information on the image analysis processing function from the storage unit 404 based on the setting data received from the imaging device 110, and writing the information to the arithmetic processing unit 412. The information on the image analysis processing function is, for example, a setting parameter indicating the order and type of calculations to be processed in the arithmetic processing unit 412, coefficients of the calculations, and the like. The arithmetic processing unit 412 includes a plurality of arithmetic circuits required to execute the image analysis processing function. The arithmetic processing unit 412 executes each arithmetic processing based on the information of the image analysis processing function received from the processing switching unit 411, and transmits the processing result to the imaging device 110 and/or records the processing result in the storage unit 404. In this way, the FPGA 402 extracts the setting data of the processing function to be executed, which is included in the setting data corresponding to the multiple processing functions stored in advance, and rewrites the processing content executed by the arithmetic processing unit 412 based on the extracted setting data. This allows the removable device 100 to selectively execute at least one of the multiple processing functions. In addition, by adding setting data for a newly added process as needed, the imaging device 110 can execute the latest processing. In the following, having multiple setting data corresponding to each of the multiple processing functions is expressed as having multiple processing functions. In other words, even if the FPGA 402 of the removable device 100 is configured to execute one processing function, if the processing content of the arithmetic processing unit 412 can be changed by setting data for another processing function, it is expressed as having multiple processing functions.

The SD controller 403 is a known control IC (integrated circuit) as defined in the SD standard, and controls the slave operation of the SD protocol and the reading and writing of data from and to the memory unit 404. The memory unit 404 is configured, for example, with a NAND type flash memory, and stores various information such as memory data written from the imaging device 110, information on the image analysis processing function written to the arithmetic processing unit 412, and setting data for the FPGA 402.

5 shows an example of the functional configuration of the removable device 100. The removable device 100 includes, for example, an analysis unit 501, a communication unit 502, and a storage unit 503 as its functional configuration. The analysis unit 501 executes an analysis process on an image. For example, when an analysis process setting request is input, the analysis unit 501 executes settings to make the input analysis process executable. In addition, when an image is input, the analysis unit 501 executes an analysis process set to an executable state for the input image. In this embodiment, the executable analysis processes are assumed to be a human body detection process and a face detection process, but are not limited to these. For example, it may be a process of determining whether or not a person stored in advance is included in an image (a face recognition process to be described later). For example, the degree of match between the image feature amount of a person stored in advance and the image feature amount of a person detected from an input image is calculated, and if the degree of match is equal to or greater than a threshold, it is determined that the person is a person stored in advance. In addition, for the purpose of privacy protection, it may be a process of superimposing a predetermined mask image or performing mosaic processing on a person detected from an input image. The processing may also be a process of detecting whether a person in an image is performing a specific behavior using a learning model that has learned a specific behavior of the person through machine learning. Furthermore, the processing may be a process of determining what kind of area in an image is. For example, the processing may be a process of determining what kind of area in an image is using a learning model that has learned buildings, roads, people, the sky, etc. through machine learning. As described above, the executable analysis processing can be applied to image analysis processing using machine learning and image analysis processing without machine learning. In addition, each of the above analysis processing may be performed in cooperation with the imaging device 110, rather than being performed by the removable device 100 alone. The communication unit 502 communicates with the imaging device 110 via the I/F unit 401. The storage unit 503 stores information such as the setting contents of the analysis processing and the time required for the analysis processing. The time required for the analysis processing may be, for example, the actual time required when a specific processing is executed, and may be a time that guarantees that the analysis processing will be completed by waiting for that time. In addition, the time required for the analysis processing may be the number of clock cycles of the processing in the analysis processing. Note that this time information may be stored individually for each analysis processing. In addition, the time required for the analysis process may be initially set to a sufficiently large value, and then the time required when the process is actually performed may be collected as a statistical value, and the time may be updated based on the collected value.

<Processing flow>
Next, an example of the flow of processing executed in the system will be described. Among the following processes, the processing executed by the imaging device 110 is realized, for example, by the processor in the arithmetic processing unit 203 executing a program stored in a memory or the like. However, this is only one example, and some or all of the processing described below may be realized by dedicated hardware. In addition, the processing executed by the removable device 100 and the input/output device 130 may also be realized by the processor in each device executing a program stored in a memory or the like, or some or all of the processing may be realized by dedicated hardware.

(Overall flow)
FIG. 6 shows a schematic flow of a series of image analysis processes executed by the system. In this process, first, the user attaches the removable device 100 to the imaging device 110 (S601). The imaging device 110 executes an initialization sequence for the removable device 100 (S602). In this initialization sequence, a predetermined command is transmitted and received between the imaging device 110 and the removable device 100, so that the imaging device 110 can use the removable device 100. After that, the imaging device 110 grasps the processes that can be executed by the removable device 100, and grasps the processes that can be executed locally (the processes that can be executed by the imaging device 110 alone or by a combination of the imaging device 110 and the removable device 100) (S603). Note that the removable device 100 can be configured to be able to execute any process, but processes unrelated to the processes to be executed on the imaging device 110 side may not be taken into consideration. In one example, the imaging device 110 may hold a list of processes that can be executed, which is obtained in advance from, for example, the input/output device 130. In this case, when the imaging device 110 obtains information indicating a process that the removable device 100 can execute from the removable device 100, the imaging device 110 can grasp only the processes that can be executed depending on whether the process is included in the list. Next, the imaging device 110 determines the process to be executed, and executes the setting of the removable device 100 as necessary (S604). That is, when at least a part of the process determined to be executed is executed by the removable device 100, the setting of the removable device 100 for the process is executed. In this setting, for example, the FPGA 402 may be reconfigured using the setting data corresponding to the process to be executed. Then, the imaging device 110 or the removable device 100 executes the analysis process (S605). After that, the imaging device 110 executes the post-processing (S606). Note that the processes of S605 and S606 are executed repeatedly. The processing of FIG. 6 is executed, for example, when the removable device 100 is attached, but at least a part of the processing of FIG. 6 may be executed repeatedly, for example, when the removable device 100 is removed, such that the processing of S603 is executed again.

(Process to determine executable processes)
Fig. 7 shows an example of the flow of a process for identifying processes executable by the imaging device 110. This process corresponds to the process of S603 in Fig. 6, and may be executed when a device such as the removable device 100 is attached to or removed from the imaging device 110, or when the imaging device 110 is powered on. In this process, the imaging device 110 reads out processes executable by the removable device 100, integrates them with analysis processes executable by the imaging device 110 itself, and identifies analysis processes executable by the imaging device 110.

First, the control unit 304 of the imaging device 110 reads out a first processing list, which is a list of processing that can be executed by the analysis unit 305 of the imaging device 110 itself, stored in the storage unit 303 (S701). Next, the control unit 304 determines whether the attached device is, for example, a device having only a conventional storage function, or a specific device having a specific processing function such as the removable device 100 (S702). For example, the control unit 304 controls the device communication unit 306 to issue a read request (read command) to the attached device to a specific address, and reads out flag data stored at the specific address. Then, the control unit 304 can determine whether the removable device 100 is a specific device having a specific processing function based on the read flag data. However, this is merely an example, and whether the attached device is a specific device may be determined by other methods.

If the attached device is a specified device (YES in S702), the control unit 304 executes a process to grasp the processes that can be executed in the device (removable device 100). The control unit 304 controls the device communication unit 306 to communicate with the removable device 100 and acquires a list of processes that can be executed in the removable device 100 (hereinafter referred to as the "second process list") (S703). The control unit 304 can acquire the second process list by reading data stored at a specific address, for example, in the same way as when determining whether the removable device 100 is a specified device. Note that, for example, the second process list can be stored at the same address as the flag data that determines whether the removable device is a specified device. In this case, the imaging device 110 can simultaneously acquire the flag data and the second process list by accessing the address, and simultaneously execute the process of S702 and the process of S703. However, this is not limited to this, and these data may be stored at different addresses. Thereafter, the control unit 304 creates an integrated process list that integrates the first process list of processes that can be executed by the imaging device 110 itself, which is read from the storage unit 303, and the second process list acquired from the removable device (S704), and ends the process.

This integrated process list is a list that indicates processes that can be executed locally on the imaging device 110 side, without processing being performed by a device such as a server on the network. In this embodiment, the integrated process list is a list obtained by the union of the processes included in the first process list and the processes included in the second process list, and is a list of the processes included in at least one of the first process list and the second process list. However, this is not limited to this, and for example, when a process included in the first process list and a process included in the second process list become executable, the executable process may be added to the integrated process list.

If the attached device is not the specified device (NO in S702), the control unit 304 determines that there is no processing that can be executed by the attached device. Therefore, the control unit 304 sets the first processing list of processing that can be executed by the own device, read from the storage unit 303, as an integrated processing list indicating processing that can be executed locally on the imaging device 110 side (S705), and ends the processing. Note that if the processing in FIG. 7 is executed when the device is removed, the specified device is obviously not attached, so the first processing list will be treated as the integrated processing list.

This makes it possible to list processes that can be executed locally on the imaging device 110, based on whether or not a removable device 100 capable of executing a particular process is attached to the imaging device 110. In addition, by presenting this integrated process list to the user as described below, the user can select processes that can be executed locally on the imaging device 110 by attaching the removable device 100.

(Processing for determining the content of analysis processing)
8 shows an example of a process flow in which the imaging device 110 determines the content of an analysis process. In this process, analysis processes that can be locally executed on the imaging device 110 side are presented to the user via the input/output device 130, and the input/output device 130 accepts the user's selection. Then, the imaging device 110 determines the analysis process to be executed according to information indicating the user selection accepted via the input/output device 130.

In this process, first, the input/output device 130 communicates with the imaging device 110 and requests the acquisition of the captured image, the integrated processing list, and the post-processing list (S801). As an example, the input/output device 130 requests the imaging device 110 to transmit information by transmitting a request message defined in the ONVIF standard to the imaging device 110. However, the present invention is not limited to this, and the request to transmit information may be made by other messages or the like. Based on this request, the imaging device 110 has the imaging control unit 301 capture the surrounding environment, and the control unit 304 controls the signal processing unit 302 to process the image captured by the imaging control unit 301 and acquire the captured image (S802). Note that the imaging device 110 may continue to capture the surrounding environment and acquire the captured image regardless of the presence or absence of a request. The imaging device 110 may store the captured image locally, or may transfer the captured image to another device such as a network server and store it. The control unit 304 reads out the post-processing list stored in the storage unit 303. In this embodiment, the post-processing list includes display processing and storage processing, but is not limited to this. The control unit 304 controls the network communication unit 307 to transmit the post-processing list, the integrated processing list acquired by the processing of FIG. 7, and the captured image acquired in S802 to the input/output device 130 (S803). As an example, the imaging device 110 transmits information to the input/output device 130 by transmitting a response message to the request message defined in the ONVIF standard to the input/output device 130. However, this is not limited to this, and information may be transmitted by other messages, etc. Note that only the processing to be executed may be considered here, and the request for the captured image by the input/output device 130 in S801, the acquisition of the captured image in S802, and the transmission of the captured image to the input/output device 130 in S803 may not be performed.

The input/output device 130 receives the captured image, the integrated processing list, and the post-processing list from the imaging device 110. The input/output device 130 then presents the integrated processing list and the post-processing list to the user by displaying the screen or the like (S804). At this time, the input/output device 130 may also present the captured image to the user by displaying the screen or the like. The user then checks the displayed integrated processing list and post-processing list, and selects the analysis process to be executed (hereinafter referred to as the "process to be executed") from the integrated processing list (S805). The user also selects the post-process to be executed (hereinafter referred to as the post-process to be executed) (S806). The input/output device 130 transmits information indicating the selection result of the process to be executed and the post-process to be executed to the imaging device 110 (S807).

When the control unit 304 of the imaging device 110 receives information indicating the process to be executed selected by the user from the input/output device 130 by controlling the network communication unit 307, it determines whether or not the process to be executed is included in the second process list (S808). If the process to be executed is not included in the second process list (NO in S808), the control unit 304 executes the process within the imaging device 110, and ends the process of FIG. 8 without notifying the removable device 100, etc. On the other hand, if the process to be executed is included in the second process list (YES in S808), the control unit 304 controls the device communication unit 306 to send a setting request for the process to be executed to the removable device 100 (S809).

The communication unit 502 of the removable device 100 receives a setting request for the process to be executed from the imaging device 110. The communication unit 502 outputs the setting request for the process to be executed received from the imaging device 110 to the analysis unit 501. Based on the setting request for the process to be executed input from the communication unit 502, the analysis unit 501 acquires the setting of the process to be executed from the storage unit 503 and executes the setting so that the removable device 100 can execute the process to be executed (S810). For example, after the setting process is completed, the communication unit 502 transmits a setting completion notification to the imaging device 110 (S811). Note that the communication unit 502 only needs to notify the imaging device 110 of information to prevent the imaging device 110 from writing data at a timing when the setting of the removable device 100 is not completed, and may notify the imaging device 110 of information on the setting completion timing before the setting is actually completed. The control unit 304 of the imaging device 110 controls the device communication unit 306 to receive a setting completion notification from the removable device 100.

The notification of the completion of the setting from the removable device 100 to the imaging device 110 can be executed, for example, using one of the following three methods. In the first notification method, the communication unit 502 outputs a BUSY signal when the setting of the process to be executed is not completed during the write process of the first block of data from the imaging device 110. The output of the BUSY signal is performed, for example, by driving the DATA signal line defined in the SD standard to a low state. In this case, the imaging device 110 can determine whether the setting of the process to be executed is completed by checking the BUSY signal. In the second notification method, the time until the setting of the process to be executed is completed is stored in advance in the above-mentioned specific address, and the imaging device 110 reads out the information on the time until the setting is completed. The imaging device 110 outputs the write data (issues a write command) after the time until the setting of the process to be executed has elapsed. This makes it possible for the imaging device 110 to transmit the captured image data after the setting of the process to be executed is completed. In the third notification method, the analysis unit 501 writes a setting completion flag to a second specific address of the removable device 100 when the setting of the process to be executed is completed. The imaging device 110 can determine whether the setting of the process to be executed is completed by reading the data of this second specific address. Note that the address information to which the setting completion flag is written may be stored in the specific address described above, or may be stored in a different address.

(Control of analysis process execution)
9 shows an example of a flow of control when the imaging device 110 executes an analysis process. In this process, first, the imaging control unit 301 captures the surrounding environment (S901). The control unit 304 controls the signal processing unit 302 to process the image captured by the imaging control unit 301 and acquire the captured image. After that, the control unit 304 controls the analysis unit 305 to execute pre-analysis processing on the captured image input from the control unit 304 and acquire an image of the pre-analysis processing result (S902). Then, the control unit 304 determines whether the process to be executed is included in the second process list (S903).

When the control unit 304 determines that the process to be executed is not included in the second process list (NO in S903), it controls the analysis unit 305 to execute the process to be executed on the image resulting from the pre-analysis process within the imaging device 110 (S904). Then, the control unit 304 controls the analysis unit 305 to execute post-analysis process on the analysis process result (S913), and ends the process.

If the target process is included in the second process list (YES in S903), the control unit 304 controls the device communication unit 306 to request the removable device 100 for an analysis process time, which is the time from the execution request of the target process to the completion of the process (S905). Note that the analysis process time may be referred to as the "processing time" below. Then, in response to receiving the request for the processing time, the communication unit 502 of the removable device 100 transmits the processing time information read from the storage unit 503 to the imaging device 110 (S906). Note that in this embodiment, for example, the processing time information may be stored in an address accessed by the control unit 304 in order to determine whether the removable device 100 is a specified device in S702. Note that the processing time information may be stored in another address of the storage unit 503. For example, the control unit 304 of the imaging device 110 may send a read request (read command) to the address where the processing time information is stored, and obtain the processing time information as a response thereto. Processing time information is stored for each process, and the processing time associated with the process to be executed specified by the imaging device 110 is notified to the imaging device 110.

Also, for example, the processing time may be calculated as the product N/(f×k) of the operating frequency f, the multiplication value k (for example, by a phase-locked loop (PLL)), and the number of clock cycles N from when the analysis process is requested to when it is completed. In one example, if the operating frequency f is 5 MHz, k is 2, and N is 100,000, the processing time is calculated as 10 ⁵ /(5×10 ⁶ ×2)=10 ⁻² [seconds], and 10 milliseconds. The control unit 304 may obtain information on the operating frequency by reading out information stored in the storage unit 503 or from the hardware arithmetic processing unit 203 as information indicating its performance, for example. Note that information on the operating frequency of the arithmetic processing unit 412 of the removable device 100 may be notified to the imaging device 110, for example, in S906, or, for example, when the removable device 100 is attached to the imaging device 110. Furthermore, the control unit 304 may obtain the multiplication value as information indicating the performance, for example, by reading out the information stored in the storage unit 503 or, for example, from a phase-locked loop circuit of the hardware of the removable device 100. The number of clock cycles of the analysis process may be stored in the storage unit 503 in association with the analysis process, for example, and the control unit 304 may obtain the number of clock cycles corresponding to the process to be executed from the storage unit 503. In this way, in S906, information in any format capable of identifying the processing time of the process to be executed is notified to the imaging device 110 from the removable device 100.

In addition, when one or more divided images are input to the removable device 100 and a predetermined analysis process is performed using the one or more divided images collectively, information that can specify the processing time for each divided image may be notified from the removable device 100 to the imaging device 110. For example, the removable device 100 may notify the imaging device 110 of a value T as the processing time for one divided image. In this case, the imaging device 110 may specify dT, which is the multiplication of the number d of divided images and the notified processing time T, as the time until the entire processing is completed. Note that the one or more divided images that are processed collectively may be a part or all of the image obtained by shooting. In other words, the processing may be performed collectively on one or more divided images that constitute a part of the image obtained by shooting, or on multiple divided images that constitute the entire image obtained. In this case, the number of divided images to be processed collectively is specified as the number d of divided images described above. Also, for example, when a process such as integrating the results of the processes is performed after the process for each divided image, the removable device 100 may further notify the imaging device 110 of the time T0 for the process such as integration. In this case, the imaging device 110 may specify, for example, the time until the entire process is completed as dT+T0. Also, information on the number of clock cycles for each process such as the process for each divided image and the integration may be notified. Note that, when multiple processes are performed consecutively, for example, information that can specify the processing time until each of the multiple processes is completed may be notified to the imaging device 110, or the processing time until the entire process is completed may be notified to the imaging device 110. In either case, the imaging device 110 obtains information that can specify the processing time until the entire process performed by the removable device 100 is completed. Then, the imaging device 110 performs calculations as necessary to specify the processing time.

Then, the control unit 304 controls the device communication unit 306 to transmit the analysis preprocessing result to the removable device 100 (S907). For example, the control unit 304 transmits the image of the analysis preprocessing result to the removable device 100 by issuing a write request (write command) for the analysis preprocessing result. Then, the control unit 304 controls the device communication unit 306 to transmit a request for the processing to be executed to the removable device 100 (S908). Then, in the imaging device 110, the control unit 304 stops the processing and waits for the length of the processing time specified based on the information acquired from the removable device 100 (S909). Meanwhile, in the removable device 100, the communication unit 502 outputs the image of the analysis preprocessing result received in S907 to the analysis unit 501, and the analysis unit 501 executes the processing to be executed set in S810 of FIG. 8 for the image (S910). The analysis unit 501 stores the processing results, for example, in an address assigned to each process in the storage unit 503.

After the processing time has elapsed, the control unit 304 resumes the processing. Then, the control unit 304 controls the device communication unit 306 to request the analysis processing result from the removable device 100 (S911), and the communication unit 502 of the removable device 100 notifies the imaging device 110 of information indicating the result (S912). For example, the control unit 304 controls the device communication unit 306 to issue a read request (read command) to the address where the analysis processing result is stored, and obtains the information of the processing result stored at that address. Then, the control unit 304 of the imaging device 110 controls the device communication unit 306 to receive the analysis processing result from the removable device 100. Then, the control unit 304 controls the analysis unit 305 to execute post-analysis processing on the analysis processing result (S913).

(Post-processing execution control)
10 shows an example of a flow of control when the imaging device 110 executes post-processing. In this process, the control unit 304 of the imaging device 110 determines whether the post-processing to be executed includes "display" (S1001). When the control unit 304 determines that the post-processing to be executed includes display (YES in S1001), it controls the network communication unit 307 to transmit the result of the analysis process to the input/output device 130 (S1002). When the input/output device 130 receives the result of the analysis process from the imaging device 110, it presents the result of the analysis process to the user by displaying it on a screen or the like (S1003). On the other hand, when the control unit 304 determines that the post-processing to be executed does not include display (NO in S1001), the processes of S1002 and S1003 are not executed.

The control unit 304 of the imaging device 110 also determines whether the post-processing to be executed includes "save" (S1004). The determination of S1004 may be executed before S1001 or in parallel with S1001. If the control unit 304 determines that the post-processing to be executed includes save (YES in S1004), it controls the storage unit 303 to store the results of the analysis process and terminates the process. On the other hand, if the control unit 304 determines that the post-processing to be executed does not include save (NO in S1004), it terminates the process without executing the process of S1005.

In this way, the imaging device 110 can transfer the results of the analysis process to the input/output device 130 or store them in the memory unit 303 depending on the selected post-processing, without receiving any special setting operations from the user.

In this embodiment, as described above, when the imaging device 110 causes the removable device 100 to execute part or all of the analysis process, the imaging device 110 acquires the processing time from the request to execute the process to the completion of the process (by performing calculations, etc., as necessary). Then, the imaging device 110 inputs image data (preprocessing result) for the process, requests the execution of the process, waits until the acquired processing time has elapsed, and then reads out the processing result. In this way, in a configuration in which the removable device 100 cannot automatically output the processing result after the process is completed, the imaging device 110 can acquire the processing result at an appropriate timing. As a result, for example, the processing result is read out immediately after the process is completed (or after a sufficiently small time has elapsed after the process is completed), so that the imaging device 110 and the removable device 100 can cooperate to speed up the process executed. In addition, the imaging device 110 can be prevented from attempting to acquire the processing result before the process is completed, thereby preventing unnecessary memory access.

In this embodiment, the processing time is the time from the request for processing to the completion of processing, but for example, the processing time may be the time elapsed from the timing when the data to be processed, such as the result of preprocessing on the divided image, is input as the starting point until the processing is completed. In this case, for example, the process of transmitting and receiving the request for processing in S908 does not need to be performed, and the analysis process is automatically started by inputting the data to be processed. The imaging device 110 may transmit the data to be processed to the removable device 100, wait for the processing time from the completion of the transmission, and then read the processing result. According to this, the transmission and reception of the instruction to execute the process is omitted, so that the number of times the command is transmitted and received can be reduced and the efficiency of the process can be improved. In this embodiment, it can be said that the write command for the data to be processed is used as the request for execution of the process. In other words, a command to request execution of the process may be issued separately from the write command for the target to be processed, or the write command for the target to be processed may be used as the command to request execution of the process. In addition to the timing of the command, the processing time may be the time elapsed from the timing when the process is actually started in the removable device 100 until the processing is completed. In this way, the elapsed time from a timing arbitrarily determined in association with the start of processing to the completion of processing can be considered as processing time.

In addition, in this embodiment, the imaging device 110 acquires the processing time related to the process when the removable device 100 starts the process, but this is not limited to this. For example, when the removable device 100 is attached to the imaging device 110, information that can identify this processing time may be included in the information that the imaging device 110 reads in S702 to determine whether the removable device 100 is a specified device. In this way, the information only needs to be read once when the removable device 100 is attached, thereby reducing the number of times commands are sent and received and improving processing efficiency.

In addition, in the above embodiment, image analysis processing has been used as an example of analysis processing, but the present invention can also be applied to audio analysis processing. For example, the present invention can be applied to processing that detects audio patterns such as screams, gunshots, and the sound of breaking glass. For example, audio features are extracted using various audio data analysis methods such as spectral analysis, and compared with the detected audio pattern. Then, the degree of match is calculated, making it possible to detect a specific audio pattern.

When performing voice analysis processing, the voice data is divided into voice data of a predetermined time, and the voice analysis processing is performed using the voice data of the predetermined time as a unit. This predetermined time varies as appropriate depending on the voice pattern to be detected. Therefore, voice data for each time corresponding to the voice pattern to be detected is input to the removable device 100. The removable device 100 has a function to analyze the input voice data and a function to hold the input voice data.

In the above-described embodiment, the removable device 100 capable of non-temporarily storing data input from the imaging device 110 has been described as an example. However, in some embodiments, the removable device 100 may not be capable of non-temporarily storing data input from the imaging device 110. In other words, the removable device 100 may only perform analysis processing on the data input from the imaging device 110, and may not need to non-temporarily store the data. In other words, it is possible to make the removable device 100 intended only for analysis processing, rather than for the purpose of storing data like a normal SD card.

<Modification>
In the above embodiment, a method for improving the efficiency of making the removable device 100 execute a process when the imaging device 110 can obtain the processing time in the removable device 100 in advance has been described. On the other hand, due to factors such as a voltage drop or the environment in which the imaging device 110 is placed, the actual processing time in the removable device 100 may be longer than the obtained processing time. If the actual processing time becomes longer, the imaging device 110 may attempt to obtain the processing result from the removable device 100 before the target process is completed. In contrast, in this modified example, if the actual processing time becomes longer than the obtained processing time, the processing result is obtained after the target process is completed. This method will be described below.

11 shows an example of the hardware configuration of the removable device 100 according to this embodiment. As shown in FIG. 11, in this embodiment, the arithmetic processing unit 412 includes a single or multiple arithmetic execution units 1101 and a single or multiple RAMs 1102. Each process executable by the removable device 100 may be configured, for example, by a combination of multiple partial processes. Here, for example, at least a portion of the multiple arithmetic execution units 1101 may be configured by a circuit that executes separate partial processes. Note that some of the arithmetic execution units 1101 may be configured to execute common partial processes in parallel on different data. The multiple arithmetic execution units 1101 may be connected to each other, or may be connected to a RAM for temporary recording, for example. Each partial process is associated with an index indicating the processing order (which process the partial process is) in the process to be executed, a command indicating the type of operation to be executed, and a weight indicating the coefficient of the operation to be executed. In this case, the index, command, and weight associated with the partial process are stored in the memory unit 404 and expanded in the RAM 1102. The calculation execution unit 1101 can execute the corresponding partial processing by accessing the RAM 1102 and acquiring the calculation command. However, this is not limited to this, and for example, the index of the partial processing does not have to exist. In this case, the calculation command and weights can be expanded in the RAM 1102 according to the order in which they are processed.

(Communication between the imaging device 110 and the removable device 100)
Here, communication between the imaging device 110 and the removable device 100 will be described. The arithmetic processing unit 203 of the imaging device 110 and the SD controller 403 of the removable device 100 are connected by a power line, a GND line, a clock line, a command line, and a data line via a device insertion socket of the SD I/F unit 205 of the imaging device 110. The clock line, the command line, and the data line are connected via the FPGA 402. The clock line communicates a clock for synchronization output from the arithmetic processing unit 203. The command line communicates a command issued from the arithmetic processing unit 203 to request an operation from the SD controller 403, and a response to the command from the SD controller 403 to the arithmetic processing unit 203. The data line communicates write data from the arithmetic processing unit 203 and read data from the removable device 100. Furthermore, the arithmetic processing unit 203 can determine whether the removable device 100 is inserted by determining whether the device detect signal of the device insertion socket of the SD I/F unit 205 is High or Low.

After supplying power to the SD controller 403, the arithmetic processing unit 203 issues a command on the command line. Then, in response to receiving a response from the SD controller 403 and output data indicating device information as an SD card, the arithmetic processing unit 203 sets the voltage for data communication, communication speed (clock frequency), and the like.

Figure 12 shows an example of the configuration of commands and responses communicated on a command line. These commands and responses have a configuration that conforms to the SD standard. A command 1201 issued from the arithmetic processing unit 203 to the SD controller 403 includes a command number section 1204, a command argument section 1205, and an error correction data section 1206. A value indicating the type of command is written in the command number section 1204. For example, if the value "23" is stored in the command number section 1204, it is indicated that the command is a block number designation command that designates the number of data blocks. Also, if the value "25" is stored in the command number section 1204, it is indicated that the command is a multi-write command, and if the value "12" is stored in the command number section 1204, it is indicated that the command is a data transfer stop command. In the command argument section 1205, information such as the number of transfer data blocks and memory write/read addresses is specified according to the type of command. In addition, a command start bit 1202 indicating the start position of the command is added to the first bit of the command, and a command end bit 1207 indicating the end of the command is added to the last bit of the command. In addition, a direction bit 1203 indicating that the signal is output from the imaging device 110 to the removable device 100 is also added after the command start bit 1202.

The response 1211 returned from the SD controller 403 in response to a command from the arithmetic processing unit 203 includes a response number section 1214 indicating which command the response is for, a response argument section 1215, and an error correction data section 1216. A response start bit 1212 indicating the start position of the response is added to the first bit of the response, and a response end bit 1217 indicating the end position of the response is added to the last bit of the response. A direction bit 1213 indicating that the signal is output from the removable device 100 to the imaging device 110 is also added after the response start bit 1212. The response argument section 1215 stores information such as the status of the SD card according to the command type.

Next, a method for transmitting and receiving data between the arithmetic processing unit 203 and the removable device 100 will be described. In the SD I/F unit 205, data is transferred in blocks for both writing and reading data.

There are two methods for the arithmetic processing unit 203 to transfer multiple blocks of data to the removable device 100. In the first method, the number of blocks is specified by a block number specification command for the transfer data, and then the specified number of blocks of data is transferred by the multi-write command. In the block number specification command, the number of blocks of the write data is specified in the command argument part 1205, and in the multi-write command, the address of the storage unit 404 to which the data should be written is specified in the command argument part 1205. In the second method, the data transfer is started by issuing a multi-write command without issuing a block number specification command, and the process ends when the data transfer is completed by issuing a transfer stop command. At this time, only the address of the storage unit 404 to which the data should be written is specified in the command argument part 1205 of the multi-write command. The arithmetic processing unit 203 can arbitrarily switch between the two writing methods.

When storage processing is performed, the FPGA 402 inputs the command and data sent from the arithmetic processing unit 203 directly to the SD controller 403, and the SD controller 403 stores the received data at an address in the storage unit 404 specified by the command. When image analysis processing is performed, the FPGA 402 executes analysis processing on the data sent from the arithmetic processing unit 203, and outputs the data resulting from the processing and information specifying a specific address in the storage unit 404 to the SD controller 403. The SD controller 403 stores the processing result at the specified address in the storage unit.

The arithmetic processing unit 203 has two methods for reading multiple blocks of data from the removable device 100. In the first method, the number of blocks is specified by a block number specification command, and then a multi-read command is issued, and data is read out for the specified number of blocks. In the block number specification command, the number of blocks of read data is specified in the command argument section 1205, and the memory address from which the data is read is specified in the command argument section 1205 of the multi-read command. In the second method, data reading is started by issuing a multi-read command without issuing a block number specification command, and processing ends by issuing a transfer stop command. The arithmetic processing unit 203 can arbitrarily switch between the two read methods.

When the write data or read data is one block, a single write command or single read command can be issued, allowing data to be written or read without issuing a block count specification command or a transfer stop command. In the single write command and single read command, the address of the storage unit 404 to be accessed is specified in the command argument section 1205, as in the above description.

The arithmetic processing unit 203 can transmit data to be subjected to storage processing or image analysis processing to the removable device 100 by writing to the removable device 100. In addition, the arithmetic processing unit 203 can obtain image data stored in the storage unit 404, the results of image analysis processing, and information on the image analysis processing function possessed by the removable device 100 by reading from the removable device 100.

In this modified example, when the imaging device 110 requests the removable device 100 for a processing result in S911, even if the processing is not complete, the removable device 100 efficiently notifies the imaging device 110 of the processing result. In this process, if the removable device 100 receives a request for the processing result and the processing is not complete, the removable device 100 notifies the imaging device 110 of the processing status, and the imaging device 110 waits for a certain period of time depending on the processing status before requesting the processing result again. When the processing is not complete, the waiting time is set taking into consideration the processing status of the removable device 100, so that it is possible to prevent unnecessarily long wait times while reducing the probability that the processing will be incomplete when the processing result is next requested. An example of the flow of this process is described below with reference to FIG. 13.

In this process, first, the control unit 304 of the imaging device 110 controls the device communication unit 306 to request the processing result from the removable device 100 (S1301). When the removable device 100 receives this processing result using the communication unit 502, it determines whether the analysis unit 501 has completed the processing to be executed (S1302). If the removable device 100 determines that the analysis unit 501 has completed the processing to be executed (YES in S1302), it notifies the imaging device 110 of the processing result using the communication unit 502 (S1303) and ends the process. On the other hand, if the removable device 100 determines that the analysis unit 501 has not completed the processing to be executed (NO in S1302), it notifies the imaging device 110 of the processing status information indicating the execution status of the processing by the analysis unit 501 using the communication unit 502 (S1304). The imaging device 110 calculates the remaining processing time based on the processing status information acquired from the removable device 100, and waits for the remaining processing time (S1305). Then, after the remaining processing time has elapsed, the imaging device 110 again requests the processing result from the removable device 100 (S1301). At this point, the time expected to be required from the processing status notified in S1304 to processing completion has elapsed. Therefore, by re-requesting the processing result, the imaging device 110 can acquire the processing result from the removable device 100 with a sufficiently high probability.

The processing status information may be, for example, an index of the partial process being executed by the analysis unit 501. However, the processing status information is not limited to this, and may be information in another format capable of identifying the progress of the process. For example, since information related to the partial process is expanded in the RAM 1102, the address value of the RAM 1102 accessed by the calculation execution unit 1101 may be used as the processing status information. Also, for example, the processing status information may be information indicating whether the processing to be executed has been completed. Also, the processing status information may include, for example, a value indicating the remaining processing time. In other words, the removable device 100 may be configured to be able to identify the remaining processing time of the process currently being executed, and the removable device 100 may notify the imaging device 110 of the identified remaining processing time as the processing status information.

The notification of the processing status by the removable device 100 of this modified example may be performed using a portion of the data area of the response to the request command for the processing result that is reserved in the SD standard. However, this is not limited to this, and for example, in the response on the data line to the request command for the processing result, information on the processing status may be included before the information indicating the processing result. In other words, the processing status information may be transmitted as a response to the request command for the processing result regardless of whether the processing is completed or not. In this case, in the data transmitted from the removable device 100, an area in which the processing result is stored is prepared in advance as an area of a predetermined size. Then, the imaging device 110 may treat the data of a predetermined size from the end of the data received from the removable device 100 as the processing result, and treat the other data as the processing status. In addition, in the response on the data line to the request command for the processing result, information on the processing status may be included after the information indicating the processing result. Note that the area in which the processing result is stored may contain invalid values such as all zeros if the processing is not completed. Furthermore, the imaging device 110 may treat, among the data received from the removable device 100, data before a specific pattern as the processing status, and treat data after the specific pattern as the processing result.

In addition, in this modification, the imaging device 110 may calculate the remaining processing time based on the index of the partial process being executed by the analysis unit 501, the total number of partial processes, and the elapsed time since the start of the process. For example, assume that there are N partial processes, and that the N partial processes are assigned in order of execution with indexes 0 to N-1. That is, the partial process with index 0 is executed first, then the partial process with index 1 is executed, then the partial process with index 2 is executed, and finally the process with index N-1 is executed. At this time, assume that index n is notified as information on the processing state. In this case, the partial process with index n is also being executed, and the n partial processes with indexes 0 to n-1 are in a completed state. Here, if the elapsed time is T, since n partial processes have been completed for the elapsed time T, the expected value of the time required for each partial process is T/n. Since the total number of partial processes is N, it is estimated that N×(T/n) is the time until all partial processes are completed. Then, the imaging device 110 may calculate the remaining processing time as (N/n-1)×T by subtracting the elapsed time T from the start of processing from the time until all of the partial processes are completed. Here, the removable device 100 stores the total number of partial processes, and the imaging device 110 may acquire information on the total number of partial processes from the removable device 100. Note that the remaining processing time may be calculated by, for example, having the calculation execution unit 1101 in the removable device 100 access the RAM 1102 to execute the calculation as described above, and having the imaging device 110 acquire the result of the calculation from the removable device 100.

In addition, when the information on the processing state is the address value of the RAM 1102 accessed by the calculation execution unit 1101, the remaining processing time can be specified based on the address value of the RAM 1102 accessed by the calculation execution unit 1101. For example, the range of the storage unit 404 in which the data of the processing to be executed is stored is determined in advance, and the information on the first partial processing is stored in a certain range (for example, a range of size S) from the top address within that range. Then, the next processing is stored in a certain range (range of size S) immediately after the range in which the information on the first partial processing is stored, the next processing is stored immediately after that range, and ... the information on the last partial processing is stored in a certain range at the very end of that range. In this way, the size of the data of the processing to be executed can correspond to the range of the storage unit 404 in which the data is stored. At this time, for example, the top address is A, the size of the data of the processing to be executed is K x S, and A + k x S is specified as the information on the processing state. In this case, it can be specified that there are K total partial processings, and up to the kth partial processing (index is k-1) has been completed. Therefore, if the elapsed time after the start of processing is T, the remaining processing time is (K/k-1) x T, as in the above case. This calculation is equivalent to the address-based calculation ((K x S)/(k x S)-1) x T. Note that k x S can be specified as the value obtained by subtracting the top address A of the range in which data related to the processing to be executed is stored from the address A + k x S notified as processing status information. In this way, the remaining processing time can be calculated based on address information that specifies the area in which information associated with the currently executing partial processing is stored.

The remaining processing time may be determined, for example, as the product of the elapsed time T and a certain coefficient, regardless of the processing state. Here, this coefficient is preset to, for example, less than 1. Note that in each of the above-described methods for calculating the remaining processing time, the processing time notified to the imaging device 110 from the removable device 100 in S906 may be used instead of the elapsed time T.

In this modified example, as described above, when the imaging device 110 requests the removable device 100 for a processing result, the removable device 100 notifies the imaging device 110 of the processing result if the processing is completed, or of the processing status information if the processing is not completed. Then, when the processing status information indicates that the processing is incomplete, the imaging device 110 identifies the remaining processing time and waits for the remaining processing time. Then, after waiting for the remaining processing time, the imaging device 110 executes the request for the processing result again. This allows the imaging device 110 to obtain the processing result at an appropriate timing in a configuration in which the removable device 100 cannot automatically (actively) output the processing result after the processing is completed. This allows the imaging device 110 to read the processing result immediately after the processing is completed (or after a sufficiently small time has elapsed after the processing is completed), for example, and therefore speeds up the processing executed by the imaging device 110 and the removable device 100 in cooperation with each other.

In this modified example, when the processing to be performed is not completed, the imaging device 110 is notified of the processing status information when the imaging device 110 requests the analysis processing result from the removable device 100, but this is not limited to the above. For example, the imaging device 110 may request the processing status information before requesting the analysis processing result from the removable device 100. When the processing status information is requested, the removable device 100 notifies the imaging device 110 of the processing status information. Thereafter, when the processing status information indicates that the processing to be performed is completed, the imaging device 110 requests the processing result from the removable device 100, and the removable device 100 notifies the imaging device 110 of the processing result. Also, when the processing status information indicates that the processing to be performed is not completed, the imaging device 110 waits for a period calculated as described above, for example. After waiting, the imaging device 110 can again request the processing status from the removable device 100, and the removable device 100 can again notify the imaging device 110 of the processing status.

Note that the imaging device 110 waits for the calculated time and then repeatedly requests the processing result unless it acquires processing status information indicating that the processing to be executed has been completed. However, this is not limited to the above, and the imaging device 110 may determine that an error has occurred, for example, when processing has not progressed for a predetermined period of time. For example, the imaging device 110 may determine that an error has occurred when there is no change in the index indicating the partial processing being executed multiple times in succession. Also, the imaging device 110 may determine that an error has occurred, for example, when the length of the waiting time calculated as described above exceeds a predetermined length of time. This makes it possible to identify that the imaging device 110 has fallen into an error state when the removable device 100 is in a state where processing does not end, such as when the processing is looping.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100a-100d: removable device, 110a-110d: imaging device, 120: network, 130: input/output device, 201: imaging unit, 202: image processing unit, 203: arithmetic processing unit, 204: distribution unit, 205: SD I/F unit, 301: imaging control unit, 302: signal processing unit, 303: storage unit, 304: control unit, 305: analysis unit, 306: device communication unit, 307: network communication unit, 401: I/F unit, 402: FPGA, 403: SD controller, 404: storage unit, 501: analysis unit, 502: communication unit, 503: storage unit

Claims (27)

An imaging device having a mounting part to which a device having a function of performing image analysis processing on a captured image can be attached,
An imaging device characterized in having an acquisition means for acquiring information including the number of clock cycles from the start of the image analysis process to its completion from a device attached to the mounting portion in order to calculate a processing time, which is the time from a predetermined timing associated with the start of the image analysis process by the device attached to the mounting portion to the completion of the process, calculating the processing time based on the information, and, when the device is caused to start executing the image analysis process, acquiring the results of the image analysis process from the device after the processing time has elapsed from the predetermined timing associated with the start of the image analysis process. 2. The image pickup apparatus according to claim 1 , wherein the acquisition means calculates the processing time from the number of clock cycles and a hardware performance of the device . The imaging device described in claim 1 or 2, characterized in that when the device performs the image analysis processing collectively on one or more divided images obtained by dividing the image, the acquisition means acquires the information about one divided image from the device and acquires the processing time based on the information and the number of divided images to be processed collectively . 4. The imaging apparatus according to claim 1, wherein the acquisition unit acquires the information for each process that can be executed by the device. 5. The imaging apparatus according to claim 1, wherein the predetermined timing is a timing at which a command requesting execution of the image analysis process is transmitted to the device. 5. The imaging apparatus according to claim 1, wherein the predetermined timing is a timing at which the image analysis process actually starts. The imaging apparatus according to claim 1 , wherein the acquisition means further acquires processing status information indicating an execution status of the image analysis processing. 8. The imaging apparatus according to claim 7 , wherein the acquisition means acquires the result when the processing status information indicates that the image analysis processing is completed. 9. The imaging apparatus according to claim 7 , wherein the information on the processing state is information indicating a progress status of the image analysis processing. 10. The imaging apparatus according to claim 9 , further comprising: a processor for acquiring a remaining processing time until the image analysis processing is completed from the progress status. The imaging device according to claim 10 , wherein the progress is indicated by an address of a memory being accessed by the image analysis process. 20. The imaging device according to claim 19 , wherein the remaining processing time is calculated based on the address and an area of the memory used by the image analysis process. The image analysis process is composed of a plurality of sub-processes,
The imaging device according to claim 10 , wherein the progress status is indicated by which of the plurality of partial processes is being executed. The imaging device according to claim 13 , wherein the remaining processing time is calculated based on the number of the plurality of partial processes and an order in which the partial processes being executed are executed in the image analysis processing. The image analysis process is composed of a plurality of sub-processes,
Information associated with each of the plurality of partial processes is stored in a memory;
The imaging device according to claim 10 , wherein the progress is indicated by an address of the memory being accessed by the image analysis process. 16. The imaging device according to claim 15 , wherein the remaining processing time is calculated based on the number of the plurality of partial processes and an order in which the partial processes corresponding to the addresses are executed in the image analysis process. 17. The imaging device according to claim 10 , wherein the progress status indicates whether the image analysis process is completed or not. The imaging device according to claim 17 , wherein, when the image analysis process is not completed, the remaining processing time is calculated based on the processing time. 11. The imaging apparatus according to claim 7 , wherein the information on the processing state indicates a remaining processing time until the image analysis processing is completed. The imaging device according to claim 10 , wherein the acquisition means acquires the result after waiting for the remaining processing time. 8. The imaging device according to claim 7, wherein the acquisition means acquires the result when the image analysis processing is completed, and acquires information on the processing status when the image analysis processing is not completed. 22. The imaging device according to claim 7 , wherein the acquisition unit acquires the information on the processing state by using a response in accordance with the SD standard. 8. The imaging apparatus according to claim 7 , wherein the acquiring means acquires the information on the processing state and the result by using a data line in the SD standard. 3. The imaging device according to claim 2, wherein the acquiring means acquires the information on the processing state from the data acquired from the data line based on a size in which the result is stored in the data. 3. The imaging device according to claim 2 , wherein the acquiring means acquires the information on the processing state from the data acquired from the data line based on a predetermined pattern contained in the data. A control method executed by an imaging device having a removable mounting part for a device having a function of performing image analysis processing on a captured image, comprising:
an acquisition step of acquiring information including the number of clock cycles from the start of the image analysis process to the completion of the image analysis process by a device attached to the attachment part, calculating the processing time based on the information, and acquiring a result of the image analysis process from the device after the processing time has elapsed from the predetermined timing associated with the start of the image analysis process when the device is caused to start executing the image analysis process;
A control method comprising: A computer is provided in an imaging device having a removable mounting part for a device having a function of performing image analysis processing on a captured image,
In order to calculate a processing time, which is a time from a predetermined timing associated with the start of the image analysis process by the device attached to the attachment portion to the completion of the process, information including a number of clock cycles from the start of the image analysis process to the completion of the process is acquired from the device, and the processing time is calculated based on the information;
When the device is caused to start execution of the image analysis process, a result of the image analysis process is acquired from the device after the processing time has elapsed from the predetermined timing related to the start of the image analysis process.
Program for.

Images