JP2022123734A - Image judgment method, image judgment system and image judgment program - Google Patents

Abstract

[Problem] To provide an image judgment method, an image judgment system, and an image judgment program for efficiently improving judgment accuracy. [Solution] The image judgment method includes a judgment step S107 in which an image judgment device acquires an image of a product on a production line and performs classification judgment using a judgment model that is a deep learning network, a first analysis step S111 in which a learning device analyzes the judgment result of the judgment model by extracting and visualizing a focus point of the judgment model for an image when the image is input to the judgment model, a second analysis step S112 in which a feature amount of the image in a high-dimensional space when the image is input to the judgment model is converted into a low-dimensional representation by dimensional compression and visualized, and a learning step S104 in which a judgment model is learned based on the image classified based on the analysis results in the first analysis step S111 and the second analysis step S112. [Selected Figure] Fig. 12

Description

The disclosed embodiments relate to an image determination method, an image determination system, and an image determination program.

Conventionally, in the field of AI (Artificial Intelligence), there has been known a technique for classifying objects in an image by image determination using a deep learning network such as a CNN (Convolutional Neural Network) as a determination model (for example, Patent Document 1).

By using such technology, for example, from the image of a product manufactured on a production line, it is possible to classify whether the product is good or defective, and if it is a defective product, what kind of defect it has. can be done.

JP 2018-022484 A

However, the conventional technology described above has room for further improvement in terms of efficiently improving the determination accuracy.

For example, a deep learning network is a kind of function, a black box. For this reason, in the conventional technology, even if an attempt is made to reclassify the learning images and relearn the determination model in order to improve the determination accuracy when an erroneous determination occurs, the basis for the determination is unclear in the first place. , it was difficult to properly reclassify the training images.

In addition, in order to improve determination accuracy, it is desirable to learn a determination model using a large number of learning images and to verify the determination model using a large amount of verification images. Most of the operations had to be performed visually by a person, which was complicated.

One aspect of the embodiments has been made in view of the above, and an object thereof is to provide an image determination method, an image determination system, and an image determination program capable of efficiently improving determination accuracy.

An image determination method according to an aspect of an embodiment includes a determination process, a first analysis process, a second analysis process, and a learning process. In the determination step, an image of the product on the production line is acquired, and a classification determination is performed using a determination model that is a deep learning network. The first analysis step analyzes the determination result of the determination model by extracting and visualizing the points of interest of the determination model with respect to the image when the image is input to the determination model. In the second analysis step, when the image is input to the judgment model, the feature amount of the image in a high-dimensional space is converted into a low-dimensional representation by dimensional compression and visualized, thereby making the judgment result of the judgment model. to parse The learning step learns the determination model based on the images classified based on the analysis results in the first analysis step and the second analysis step.

According to one aspect of the embodiment, it is possible to efficiently improve the determination accuracy.

FIG. 1 is a schematic explanatory diagram (Part 1) of an image determination method according to an embodiment. FIG. 2 is a schematic explanatory diagram (Part 2) of the image determination method according to the embodiment. FIG. 3 is a block diagram of the learning device according to the embodiment. FIG. 4 is a block diagram of the analysis unit. FIG. 5 is an explanatory diagram (Part 1) of a specific example of visualization by the point-of-interest extraction unit. FIG. 6 is an explanatory diagram (part 2) of a specific example of visualization by the point-of-interest extraction unit. FIG. 7 is an explanatory diagram (part 1) of a specific example of visualization by the dimension compression unit. FIG. 8 is an explanatory diagram (part 2) of a specific example of visualization by the dimension compression unit. FIG. 9 is an explanatory diagram (part 3) of a specific example of visualization by the dimension compression unit. FIG. 10 is a block diagram of the image determination device according to the embodiment. FIG. 11 is a block diagram of the projector control device according to the embodiment. FIG. 12 is a processing sequence showing a processing procedure executed by the image determination device 100 according to the embodiment. FIG. 13 is a hardware configuration diagram showing an example of a computer that implements the functions of the learning device.

Hereinafter, embodiments of an image determination method, an image determination system, and an image determination program disclosed in the present application will be described in detail with reference to the accompanying drawings. In addition, this invention is not limited by embodiment shown below.

First, an outline of an image determination method according to an embodiment will be described with reference to FIGS. 1 and 2. FIG. FIG. 1 is a schematic explanatory diagram (Part 1) of an image determination method according to an embodiment. FIG. 2 is a schematic explanatory diagram (part 2) of the image determination method according to the embodiment.

In the following description, round cookies are manufactured as a product on a manufacturing line, and defective products with chips, scorches, cracks, etc. are detected in pre-shipment inspections of such cookies. In the following description, it is assumed that the determination model for image determination is a deep learning network.

As shown in FIG. 1, the image determination system 1 according to the embodiment includes a learning device 10, an image determination device 100, and a projector control device 200. FIG.

The image determination device 100 and the projector control device 200 are devices corresponding to an edge platform in so-called edge computing, and are provided in a production line including a camera 150, a conveyor device 300, a projector 400 (see FIG. 2), and the like.

The learning device 10 is provided so as to be able to communicate with the manufacturing line via a network N such as an intranet, the Internet, or a mobile phone network. The main functions of the learning device 10 are, for example, collecting learning images from a manufacturing line, classifying the collected learning images to generate a learning data set, and learning a judgment model by deep learning using the data set. (function F1). Also, the learning device 10 distributes the learned determination model to the image determination device 100 via the network N. FIG.

Image determination device 100 acquires images of cookies P1, P2, P3, . In response, the determination result is output (function F2).

For example, FIG. 1 shows an example in which the image determination apparatus 100 determines that the cookie P1 is defective with "missing" and the cookie P2 is defective with "burnt" through image determination. Note that the determination result includes at least the determined classification class of the image and its score (similarity, accuracy, etc.).

In addition, the determination result for the learning device 10 includes the actually determined image as the learning image. The learning device 10 verifies the determination results manually, for example, by an operator (equivalent to an example of a “user”). to relearn the decision model.

By repeating such feedback, the image determination system 1 can improve the determination accuracy of the determination model.

By the way, as already mentioned, a deep learning network is, so to speak, a kind of function, a black box. For this reason, conventionally, when an erroneous judgment occurs, even if it is desired to reclassify the training images and re-learn the judgment model, the basis for the judgment is unclear in the first place. It was difficult to categorize.

In addition, in order to improve the accuracy of determination, it is desirable to learn a determination model using a large number of training images and to verify the determination model using a large amount of verification images. Many of them had to be visually checked by an operator, which was troublesome.

Therefore, in the image determination method according to the embodiment, an image of a product in a manufacturing line is acquired, a classification determination is performed using a determination model that is a deep learning network, and the image is determined when the image is input to the determination model. Analyzing the judgment result of the judgment model by extracting and visualizing the points of interest of the model, and converting the feature amount of the image in the high-dimensional space when the image is input to the judgment model into a low-dimensional representation by dimensional compression. The judgment model is analyzed by visualizing it as a model, and the judgment model is learned based on the images classified based on the analysis results obtained by extracting the point of interest and dimensionality reduction.

Specifically, as shown in FIG. 1, in the image determination method according to the embodiment, the learning device 10 analyzes the determination result of the determination model using two methods (step S1). In the first technique, the learning device 10 visualizes the basis of judgment by the judgment model by "focus point extraction" using a gradient-weighted class activation mapping technique (Grad-CAM).

As a result, the operator can grasp at a glance "where the judgment model was classified (judgment)". It becomes possible. A specific example of visualization by the first method will be described later with reference to FIGS. 5 and 6. FIG.

In the second method, the learning device 10 visualizes the determination result in a more easily viewable form by “dimensionality reduction” (also referred to as dimensionality reduction) using UMAP (Uniform Manifold Approximation and Projection).

UMAP is a non-linear dimensionality compression technique based on machine learning, based on Riemannian geometry and algebraic topology, which preserves the data structure of high-dimensional space and converts it to low-dimensional data while minimizing cross-entropy between topologies. That is, in the image determination method according to the embodiment, UMAP is used to convert the feature amount of the image in the high-dimensional space when the image is input to the determination model into a low-dimensional expression by dimensional compression and visualize it.

For this reason, according to the second method, it is possible to express the distribution of judgment results in a low-dimensional embedding space in a form with less variation. Visualization becomes possible. A specific example of visualization by the second method will be described later with reference to FIGS. 7 to 9. FIG.

Then, the learning device 10 reclassifies and re-learns the learning images based on the analysis results obtained by these two methods (step S2), and delivers the re-learned determination model to the image determination device 100. FIG. Then, the image determination apparatus 100 performs subsequent image determination using the re-learned determination model.

Therefore, according to the image determination method according to the embodiment, it is possible to efficiently improve the determination accuracy.

On the other hand, the projector control device 200 is a device that controls the projector 400 provided in the manufacturing line. Specifically, as shown in FIG. 2 , in the image determination method according to the embodiment, the projector control device 200 performs projector projection according to the determination result of the image determination device 100 .

More specifically, projector control device 200 causes projector 400 to project markers onto cookies P1, P2, P3, . . . flowing on conveyor device 300 (step S3).

At this time, the projector control device 200 scrolls the marker at the same speed as the conveying speed of the conveyor of the conveyor device 300 (step S31). In other words, projector controller 200 causes projector 400 to project a marker such that the marker tracks target cookies P1, P2, P3, .

Also, the projector control device 200 changes the color and shape of the marker according to the classification class and the action to be taken (step S32). For example, the projector control device 200 changes the color and shape of the marker between the cookie P1 with the classification class of "missing" and the cookie P2 with the classification class of "burnt".

Also, for example, the projector control device 200 changes the color or shape of the marker according to the action to be taken, such as "should be removed from the conveyor" or "should be fed back to production." In addition, according to the scores included in the determination result, the projector control device 200 projects a recognizable marker on "what the AI struggled to determine", that is, on the score in the gray zone, and the line manager A visual check may be prompted.

Thus, according to the image determination method according to the embodiment, it is possible to appropriately reflect the result on the production line according to the high determination accuracy of the determination model. Note that the projector control device 200 is an example of a “determination result reflecting device”. Therefore, the determination result reflection device may be another device that is provided in the manufacturing line and that should reflect the determination result of the image determination device 100 . For example, the determination result reflection device may be a cookie baking device or the like that adjusts the heat level according to the determination result of the image determination device 100 .

The configuration of the image determination system 1 to which the image determination method according to the above-described embodiment is applied will now be described more specifically.

FIG. 3 is a block diagram of the learning device 10 according to the embodiment. Moreover, FIG. 4 is a block diagram of the analysis part 13d. 3, 4, and FIGS. 10 and 11 shown later, constituent elements necessary for explaining the features of this embodiment are represented by functional blocks, and descriptions of general constituent elements are omitted. is doing.

In other words, each component illustrated in FIGS. 3, 4, 10 and 11 is functionally conceptual and does not necessarily need to be physically configured as illustrated. For example, the specific forms of distribution and integration of each functional block are not limited to those shown in the figure, and all or part of them can be functionally or physically distributed in arbitrary units according to various loads and usage conditions.・It is possible to integrate and configure.

In the explanation using FIGS. 3, 4, 10 and 11, the explanation of the components already described may be simplified or omitted in some cases.

As shown in FIG. 3 , the learning device 10 according to the embodiment includes a communication section 11 , a storage section 12 and a control section 13 . Further, the learning device 10 is connected to the operation unit 3 and the display unit 5 . The operation unit 3 is implemented by a keyboard, mouse, touch panel, or the like. The display unit 5 is realized by a display or the like.

The communication unit 11 is implemented by, for example, a NIC (Network Interface Card) or the like. The communication unit 11 is wired or wirelessly connected to the network N, and transmits and receives information to and from the manufacturing line including the image determination device 100 .

The storage unit 12 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, or a storage device such as a hard disk or an optical disk. In the example shown in FIG. 3, the storage unit 12 stores a collected information database (DB) 12a, a learning data set 12b, and a determination model 12c.

The collected information database 12a is a database in which various data including determination results collected by a collection unit 13a (to be described later) via the communication unit 11 are stored. The learning data set 12b is a collection of learning images classified into classification classes by the classification unit 13b, which will be described later, based on the determination results stored in the collected information database 12a and the operator's operation via the operation unit 3. is a dataset. The judgment model 12c is a deep learning network learned by a learning unit 13c, which will be described later.

The control unit 13 is a controller, and various programs (not shown) stored in the storage unit 12 are executed by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like, using the RAM as a work area. It is realized by being Also, the control unit 13 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 13 includes a collection unit 13a, a classification unit 13b, a learning unit 13c, an analysis unit 13d, a display control unit 13e, and a distribution unit 13f. Realize or carry out.

The collection unit 13 a collects determination results from the image determination device 100 via the communication unit 11 . The collecting unit 13a also stores the collected determination results in the collected information database 12a.

Based on the determination result stored in the collected information database 12a and the operator's operation via the operation unit 3, the classification unit 13b classifies the learning image into "normal", "missing", "burnt", and "cracked". are classified for each classification class, and a learning data set 12b is generated. The learning unit 13c learns the judgment model 12c based on the learning data set 12b.

The analysis unit 13d analyzes the determination result by the above-described first method and second method based on the determination result stored in the collected information database 12a and the determination model 12c. As shown in FIG. 4, the analysis unit 13d has a point-of-interest extraction unit 13da and a dimension compression unit 13db.

The point-of-interest extraction unit 13da extracts the point of interest using Grad-CAM as described above, and visualizes the basis for determination by the determination model 12c when the learning image is input to the determination model 12c.

The dimension compression unit 13db visualizes the distribution of the determination results in the low-dimensional embedding space by the dimension compression using UMAP described above. Note that UMAP is an example of a dimension compression method, and the use of other methods is not limited. For example, principal component analysis, t-distributed stochastic neighbor embedding (t-SNE), etc. may be used, but UMAP is faster in terms of calculation speed.

Here, specific examples of visualization by the point-of-interest extraction unit 13da and the dimension compression unit 13db will be described with reference to FIGS. 5 to 9. FIG. FIG. 5 is an explanatory diagram (Part 1) of a specific example of visualization by the point-of-interest extraction unit 13da. FIG. 6 is an explanatory diagram (part 2) of a specific example of visualization by the point-of-interest extraction unit 13da.

FIG. 7 is an explanatory diagram (part 1) of a specific example of visualization by the dimension compression unit 13db. FIG. 8 is an explanatory diagram (part 2) of a specific example of visualization by the dimension compression unit 13db. FIG. 9 is an explanatory diagram (part 3) of a specific example of visualization by the dimension compression unit 13db.

First, in the description using FIG. 6, as shown in FIG. 5, consider an image of a cookie P for which the determination model 12c determines that there is a chipped portion C. FIG. The point-of-interest extraction unit 13da inputs the images p1, p2, p3, .

A deep learning network consists of two parts: a feature extraction part that stacks many layers of convolution layers and pooling layers, and a classification part that receives the feature value output and compares it with class labels to perform supervised learning. divided. In addition, the classifier is usually composed of a fully-connected multi-layer neural network, the final layer of which is a softmax layer that converts the feature quantity into a score for each classification class.

The score is the probability (can also be referred to as similarity) or certainty that the tag of each classification class is assigned to the input image. The determination result of the determination model 12c is the classification class with the maximum score.

The point-of-interest extracting unit 13da uses Grad-CAM to identify image locations that greatly affect the score for each classification class by averaging differential coefficients (which can be called gradients), and converts them into a heat map.

FIG. 6 shows an example of such a heat map. In the example of FIG. 6, for images p1 and p2, only the portion of chipping C is heatmapped, and it can be seen that the determination model 12c just focuses on chipping C and determines that chipping C exists. Therefore, it can be seen at a glance that the images p1 and p2 are suitable as learning images for the classification class "missing".

On the other hand, for the image p3, not only the chipped portion C but also the burnt portion B are heat-mapped. In other words, it can be seen at a glance that the image p3 contains noise components as a learning image of the classification class “missing”. In such a case, such a heat map can cause the operator to exclude image p3 from training as it is not suitable as a training image for the classification class "missing". As a result, it is possible to efficiently improve the determination accuracy.

Further, as shown in FIGS. 7 to 9, the dimension compression unit 13db dimension-compresses, for example, the high-dimensional space feature quantity map of the judgment model 12c to a low dimension (here, three dimensions) to obtain a low-dimensional embedding space. Visualize the distribution of judgment results. Also, the dimension compression unit 13db generates such visualization information as a GUI (Graphic User Interface) tool that can be operated by the operator.

For example, as shown in FIG. 7, the dimension compression unit 13db generates a GUI tool having check boxes corresponding to each determination result. It is assumed that "missing" and "burning" are checked in such a GUI tool as shown in FIG.

Then, as shown in FIG. 7, the distribution in the dimensionally compressed low-dimensional space of each image determined as "missing" and each image determined as "burned" is visualized. Each circle in the low-dimensional space in the drawing corresponds to each image, and the dimension compression unit 13db generates a GUI tool so that the operator can select one of them.

Here, as indicated by the cursor Cr in the figure, there is an image that is determined to be, for example, "missing", but the feature amount is much closer to "burnt" than "missing", and the operator selects this image. shall be

Then, as shown in FIG. 8, the dimension compression unit 13db generates a GUI tool so that detailed information about the image, such as the file name of the image and the label name of the classification class, is displayed. In the case of the same figure, according to the detailed information, although the corresponding image "IMG_1001.png" is classified as "missing", its feature value is very close to "burnt". is an erroneous determination that should be determined as "burnt".

Therefore, the operator can grasp the erroneous judgment at a glance by such a GUI tool. Then, the operator causes the classification unit 13b to reclassify the corresponding image, and the learning unit 13c learns the determination model 12c, thereby improving the determination accuracy of the determination model 12c.

As shown in FIG. 9, when the "crack" check box is further checked in the GUI tool, the distribution of each "crack" image is further visualized in the low-dimensional space. Also, although not shown in FIGS. 7 to 9, the operator can arbitrarily rotate the low-dimensional space on the GUI tool by 360°, enlarge it, and reduce it.

Returning to the description of FIG. The display control unit 13e causes the display unit 5 to display the analysis result of the analysis unit 13d. As indicated by the dashed arrow from the display unit 5 to the operation unit 3, when the operator instructs reclassification based on the analysis results of the analysis unit 13d as shown in FIGS. The training images in the training data set 12b are reclassified, and the learning unit 13c learns the determination model 12c.

The distribution unit 13 f distributes the determination model 12 c learned by the learning unit 13 c to the image determination device 100 via the communication unit 11 .

Next, the configuration of the image determination device 100 will be described. FIG. 10 is a block diagram of the image determination device 100 according to the embodiment.

As shown in FIG. 10, the image determination apparatus 100 according to the embodiment includes a communication section 101, a storage section 102, and a control section 103. FIG.

The communication unit 101 is realized by, for example, a NIC, like the communication unit 11 described above. The communication unit 101 is wired or wirelessly connected to the network N, the camera 150 and the projector control device 200 and transmits and receives information to and from the learning device 10 , the camera 150 and the projector control device 200 .

The storage unit 102 is realized by, for example, a semiconductor memory device such as a RAM, a ROM, a flash memory, or a storage device such as a hard disk or an optical disk, like the storage unit 12 described above. In the example shown in FIG. 10, the storage unit 102 stores a determination model 102a. The judgment model 102a corresponds to the judgment model 12c delivered from the learning device.

The control unit 103 is a controller similar to the control unit 13 described above. Realized. Also, the control unit 103 can be implemented by an integrated circuit such as an ASIC or FPGA, like the control unit 13 described above.

The control unit 103 has an acquisition unit 103a, a determination unit 103b, and an output unit 103c, and implements or executes information processing functions and actions described below.

Acquisition unit 103a acquires judgment model 12c distributed from learning device 10 via communication unit 101, and stores it in storage unit 102 as judgment model 102a. Acquisition unit 103a acquires an image of cookie P captured by camera 150 via communication unit 101, and outputs the image to determination unit 103b.

The determination unit 103b inputs the image acquired by the acquisition unit 103a to the determination model 102a, and acquires the determination result from the determination model 102a. Further, the determination unit 103b outputs the acquired determination result to the output unit 103c.

The output unit 103 c outputs the determination result from the determination unit 103 b to the learning device 10 and the projector control device 200 via the communication unit 101 .

Next, the configuration of the projector control device 200 will be described. FIG. 11 is a block diagram of the projector control device 200 according to the embodiment.

As shown in FIG. 11, the projector control device 200 according to the embodiment includes a communication section 201, a storage section 202, and a control section 203.

The communication unit 201 is implemented by, for example, a NIC, like the communication units 11 and 101 described above. The communication unit 201 is connected to the image determination device 100 and the conveyor device 300 by wire or wirelessly, and transmits and receives information to and from the image determination device 100 and the conveyor device 300 .

The storage unit 202 is realized by, for example, a semiconductor memory device such as a RAM, ROM, flash memory, or a storage device such as a hard disk or an optical disk, like the storage units 12 and 102 described above. In the example shown in FIG. 11, the storage unit 202 stores projection setting information 202a. The projection setting information 202 a is setting information regarding the projection of the marker according to the determination result from the image determination device 100 .

The control unit 203 is a controller similar to the control units 13 and 103 described above, and various programs (not shown) stored in the storage unit 202 are executed by the CPU, MPU, etc., using the RAM as a work area. It is realized by Further, like the control units 13 and 103 described above, the control unit 203 can be implemented by an integrated circuit such as an ASIC or FPGA, for example.

The control unit 203 has an acquisition unit 203a and a projection control unit 203b, and implements or executes information processing functions and actions described below.

The acquisition unit 203a acquires the determination result output from the image determination apparatus 100 via the communication unit 201, and outputs the determination result to the projection control unit 203b. Also, the acquisition unit 203a acquires the conveying speed of the conveyor from the conveyor device 300 via the communication unit 201, and outputs it to the projection control unit 203b.

The projection control unit 203b controls projection of the marker by the projector 400 based on the determination result, the transport speed, and the projection setting information 202a acquired by the acquisition unit 203a.

Next, a processing procedure executed by the image determination system 1 according to the embodiment will be described with reference to FIG. 12 . FIG. 12 is a processing sequence showing a processing procedure executed by the image determination device 100 according to the embodiment.

As shown in FIG. 12, before the image determination system 1 is operated, the learning device 10 collects learning images (step S101). Then, the learning device 10 classifies the learning images (step S102) and generates a learning data set 12b (step S103).

Then, the learning device 10 learns the determination model 12c using the learning data set 12b (step S104), and distributes the determination model 12c to the image determination device 100 (step S105).

The image determination device 100 acquires the image captured by the camera 150 (step S106), and determines the image using the determination model 102a (step S107). Then, the determination result is output to the learning device 10 and the projector control device 200 (steps S108 and S109).

The learning device 10 collects determination results from the image determination device 100 (step S110), and performs analysis by extracting the point of interest (step S111) and analysis by dimension compression (step S112).

Then, the learning device 10 reclassifies the learning images based on those analysis results (step S113). Then, the processing from step S104 is repeated.

On the other hand, the projector control device 200 repeats projector projection according to the determination result from the image determination device 100 (step S114).

Note that the learning device 10, the image determination device 100, and the projector control device 200 according to the above-described embodiments are implemented by a computer 1000 configured as shown in FIG. 13, for example. The learning device 10 will be described as an example. FIG. 13 is a hardware configuration diagram showing an example of a computer that implements the functions of the learning device 10. As shown in FIG. Computer 1000 includes CPU 1100 , RAM 1200 , ROM 1300 , HDD (Hard Disk Drive) 1400 , communication interface (I/F) 1500 , input/output interface (I/F) 1600 , and media interface (I/F) 67 .

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400 and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started up, a program depending on the hardware of the computer 1000, and the like.

HDD 1400 stores programs executed by CPU 1100 and data used by the programs. Communication interface 1500 receives data from other devices via a communication network, sends the data to CPU 1100, and transmits data generated by CPU 1100 to other devices via the communication network.

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via an input/output interface 1600 . CPU 1100 acquires data from an input device via input/output interface 1600 . CPU 1100 also outputs the generated data to an output device via input/output interface 1600 .

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200 . CPU 1100 loads the program from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor. memory and the like.

For example, when computer 1000 functions as learning device 10 according to the embodiment, CPU 1100 of computer 1000 implements each function of control unit 13 by executing a program loaded on RAM 1200 . Data in the storage unit 12 is also stored in the HDD 1400 . CPU 1100 of computer 1000 reads and executes these programs from recording medium 1800, but as another example, these programs may be obtained from another device via a communication network.

As described above, the image determination system 1 according to the embodiment includes the determination unit 103b, the point-of-interest extraction unit 13da (corresponding to an example of the “first analysis unit”), and the dimension compression unit 13db (“second analysis unit”). (corresponding to an example of an “analysis unit”) and a learning unit 13c. The determination unit 103b acquires images of products on the production line, and classifies and determines them using a determination model that is a deep learning network. The point-of-interest extraction unit 13da analyzes the determination result of the determination model by extracting and visualizing the points of interest of the determination model with respect to the image when the image is input to the determination model. The dimension compression unit 13db analyzes the determination result of the determination model by converting the feature amount of the image in the high-dimensional space when the image is input to the determination model into a low-dimensional expression by dimensional compression and visualizing it. The learning unit 13c learns a determination model based on the images classified based on the analysis results by the point-of-interest extraction unit 13da and the dimension compression unit 13db.

Therefore, according to the image determination system 1 according to the embodiment, it is possible to efficiently improve the determination accuracy.

In the above-described embodiment, an operator's operation is required to reclassify the learning images, but the present invention is not limited to this. Reclassification may be performed.

In such a case, the classification unit 13b has an image analysis function for image analysis of a heat map or a low-dimensional space map visualized by the analysis unit 13d, for example, and reclassifies the learning images based on the image analysis result. Become.

Further, in the above-described embodiment, Grad-CAM is used as an algorithm for extracting the point of interest, but it is not limited to this. An algorithm or the like may be used.

Also, in the above-described embodiment, the product in the production line is the cookie P, but the type of the product is of course not limited.

Further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspects of the invention are not limited to the specific details and representative embodiments so shown and described. Accordingly, various changes may be made without departing from the spirit or scope of the general inventive concept defined by the appended claims and equivalents thereof.

1 image determination system 10 learning device 12c determination model 13 control unit 13a collection unit 13b classification unit 13c learning unit 13d analysis unit 13da attention point extraction unit 13db dimension compression unit 13e display control unit 13f distribution unit 100 image determination device 102a determination model 103 control Unit 103a Acquisition unit 103b Determination unit 103c Output unit 200 Projector control device 203 Control unit 203a Acquisition unit 203b Projection control unit 300 Conveyor device 400 Projector

Claims (10)

A judgment step of acquiring an image of a product in a manufacturing line and classifying it using a judgment model that is a deep learning network;
a first analysis step of analyzing a determination result of the determination model by extracting and visualizing a focus point of the determination model with respect to the image when the image is input to the determination model;
a second analysis step of analyzing the determination result of the determination model by converting the feature amount of the image in a high-dimensional space when the image is input to the determination model into a low-dimensional representation by dimensional compression and visualizing the image. When,
and a learning step of learning the judgment model based on the images classified based on the analysis results of the first analysis step and the second analysis step. The learning step includes:
wherein, in the first analysis step, when the points of interest corresponding to a plurality of different classification classes are extracted for one of the images, the image is excluded from learning images and the judgment model is learned. The image determination method according to claim 1, wherein The second analysis step includes
3. The image determination method according to claim 1, wherein the distribution of determination results of the determination model based on the low-dimensional representation is made into a GUI, and a user is allowed to classify the images via the GUI. The first analysis step includes
4. The image determination method according to claim 1, wherein the point of interest is extracted and visualized using a Grad-CAM. The second analysis step includes
The image determination method according to any one of claims 1 to 4, wherein the feature amount of the image is converted into the low-dimensional representation by the dimensional compression using UMAP and visualized. The image determination method according to any one of claims 1 to 5, further comprising a determination result reflecting step of reflecting the determination result of the determination model on the production line. The production line is
Having a projector that projects a marker onto the product on the production line,
The determination result reflection step includes:
7. The image determination method according to claim 6, wherein at least the color and shape of the marker are changed according to the classification class of the product and the action to be taken. The determination result reflection step includes:
8. The image determination method according to claim 7, wherein the marker is scrolled at the same speed as the conveying speed of the product. A determination unit that acquires an image of a product in a manufacturing line and uses a determination model that is a deep learning network to make a classification determination;
a first analysis unit that analyzes a determination result of the determination model by extracting and visualizing a focus point of the determination model with respect to the image when the image is input to the determination model;
a second analysis unit for analyzing the determination result of the determination model by converting the feature amount of the image in a high-dimensional space when the image is input to the determination model into a low-dimensional representation by dimensional compression and visualizing the image; When,
and a learning unit that learns the determination model based on the images classified based on the analysis results of the first analysis unit and the second analysis unit. A judgment procedure for acquiring images of products in the manufacturing line and classifying and judging using a judgment model that is a deep learning network;
a first analysis procedure for analyzing a determination result of the determination model by extracting and visualizing a focus point of the determination model for the image when the image is input to the determination model;
A second analysis procedure for analyzing the judgment result of the judgment model by converting the feature amount of the image in a high-dimensional space when the image is input to the judgment model into a low-dimensional representation by dimensional compression and visualizing it. When,
and a learning procedure for learning the judgment model based on the images classified based on the analysis results in the first analysis procedure and the second analysis procedure.

Images