JP7392425B2 - Learning devices, learning methods and programs - Google Patents

Description

The present disclosure relates to a learning device, a learning method, and a program.

2. Description of the Related Art In recent years, progress has been made in the development of technology for automatically inspecting the quality of a manufactured product on a manufacturing line using a photographing device and checking the quality of the product based on the obtained observation image. When a plurality of samples of both non-defective products and defective products exist, the quality of the product can be determined using a classifier obtained by performing machine learning using images of non-defective products and images of defective products. However, it is usually difficult to collect more defective products than good products. Therefore, techniques have been developed for determining quality using a learning model obtained by machine learning using a plurality of images of non-defective products (Non-Patent Documents 1 to 3). In Non-Patent Documents 1 to 3, a learning model (trained model) that extracts features of a plurality of non-defective product images is generated. During inspection, features other than defects are extracted from the observed image using a learning model, and a defect-free image is restored. Defect inspection is performed by comparing the restored image and the observed image. Note that a type of machine learning model that outputs an image is generally called a generative model, but in this specification, it is simply referred to as a learning model.

Bergmann, et al., “Improving Unsupervised Defect Segmentation by Applying Structural Similarity To Autoencoders”, arXiv:1807.02011v3, February 1, 2019. Schlegl, et al., “Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery”, arXiv:1703.05921v1, March 17, 2017. Kenta Toyota, Kazuhiro Hotta, "Automatic identification of defective locations using subspace method and robust statistics", SSII2016, IS3-22, June 10, 2016

Learning operations are performed iteratively to improve the performance of the learning model. However, increasing the number of learning operations does not necessarily improve performance, and the latest learning model may not always be optimal. Therefore, it takes time and effort to adjust the learning model.

The present disclosure has been made in view of the above problems, and its purpose is to provide a learning device, a learning method, and a program that can reduce the effort of adjusting a learning model.

According to an example of the present disclosure, a learning device includes a learning section, an evaluation section, a reception section, a recording section, and an output section. The learning unit generates a learning model used to determine the attributes of the object by performing machine learning using a learning image group including one or more learning images of the object. The evaluation unit evaluates the performance of the learning model by inputting one or more evaluation images included in the evaluation image group to the learning model. The reception unit receives a first operation for updating the learning image group. The recording unit records, for each first operation, the content of the first operation and the result of evaluating the learning model generated using the updated learning image group in association with each other. The output unit outputs the history information recorded by the recording unit.

According to this disclosure, by checking the history information recorded by the recording unit, the user can determine the performance of the learning model at which timing among the learning models corresponding to each of a plurality of first operations performed in the past. You can easily figure out which is the highest. Thereby, the user can adjust the learning model as appropriate while checking the performance of the learning model for each first operation. As a result, the effort required to adjust the learning model can be reduced.

In the above-mentioned disclosure, the learning model receives an image of the target object as input, and outputs output information for determining the quality of the target object. The evaluation image group includes one or more first evaluation images in which a target object is a non-defective product, and one or more second evaluation images in which a target object is a defective product. The evaluation unit determines, as a result of the evaluation, (a) among the one or more first evaluation images, the first evaluation image in which the object is determined to be defective based on output information obtained by inputting it to the learning model; A second evaluation in which the object is determined to be a good product based on a value corresponding to the first ratio of the image and (b) output information obtained by inputting it into the learning model among the 1 or more second evaluation images. A value corresponding to the second ratio of the image for use is calculated.

According to the above disclosure, the user can understand the first ratio by checking the value according to the first ratio, and can understand the second ratio by checking the value according to the second ratio. The first ratio is the ratio at which non-defective products are determined to be defective products. The second ratio is the ratio at which defective products are determined to be non-defective products. Thereby, the user can adjust the learning model so that the first ratio and the second ratio fall within a desired range.

In the above-mentioned disclosure, the determination of the quality of the object using the output information is performed depending on whether the output information satisfies the determination condition. The evaluation unit determines the determination condition such that the second ratio is equal to or less than a predetermined threshold.

On the production line, it is necessary to avoid shipping defective products as non-defective products as much as possible. The second ratio corresponds to the ratio at which defective products are determined to be non-defective products. According to the above disclosure, the performance of the learning model is evaluated based on the pass/fail determination result performed under the determination condition that the second ratio is equal to or less than a predetermined threshold value. Thereby, under the determination condition that the second ratio is equal to or less than the threshold value, the user can adjust the learning model to a high-performance learning model.

In the above disclosure, the determination device that determines attributes of an object further includes a setting unit that configures a learning model. The recording unit further records the updated learning image group for each first operation. The setting unit sets, in the determination device, a learning model generated using a learning image group recorded in response to a first operation specified from the history information.

Alternatively, the recording unit further records the learning model generated using the updated learning image group for each first operation. The setting unit sets, in the determination device, a learning model recorded in response to a first operation specified from the history information.

According to the above disclosure, the user can easily set a learning model with high performance to the determination device while viewing the history information.

In the above disclosure, the reception unit further receives a second operation for updating the evaluation image group. The recording unit further records, for each second operation, the content of the second operation and the result of the evaluation using the updated evaluation image group in association with each other.

According to the above disclosure, the user can check the change in the evaluation result of the performance of the learning model when the evaluation image group is updated. As a result, the user can optimize the evaluation image group by checking the evaluation results.

In the above disclosure, the learning unit generates a learning model using model generation parameters. The receiving unit further receives a third operation for updating model generation parameters. The recording unit further records, for each third operation, the content of the third operation and the result of evaluation of the learning model generated using the updated model generation parameters in association with each other.

According to the above disclosure, the user can check changes in the performance evaluation results of the learning model when the model generation parameters are updated. As a result, the user can optimize the model generation parameters by checking the evaluation results.

In the above disclosure, the output unit causes the display device to display a list of first operation contents and evaluation results for each first operation.

According to the above disclosure, by viewing the list, the user can confirm changes in the performance of the learning model according to the operation.

In the above disclosure, the evaluation unit calculates an evaluation value indicating the performance of the learning model as an evaluation result. The output unit causes the display device to display a graph showing the transition of the evaluation value for each first operation.

According to the above disclosure, by viewing the graph, the user can easily confirm changes in the performance of the learning model according to the operation.

According to an example of the present disclosure, the learning method is used to determine attributes of an object by performing machine learning using a learning image group including one or more learning images in which the object is captured. A step of generating a learning model, a step of evaluating the performance of the learning model by inputting one or more evaluation images included in the evaluation image group to the learning model, and an operation of updating the learning image group. a step of accepting, a step of correlating and recording, for each operation, the content of the operation and the result of evaluation of the learning model generated using the updated training image group; and outputting the recorded history information. and a step.

According to an example of the present disclosure, a program causes a computer to execute the above setting method. These disclosures can also reduce the effort required to adjust the learning model.

According to the present disclosure, it is possible to reduce the effort required to adjust a learning model.

1 is a schematic diagram showing the overall configuration of a system according to an embodiment. 2 is a schematic diagram showing an example of the hardware configuration of the learning device shown in FIG. 1. FIG. 2 is a schematic diagram showing an example of the hardware configuration of the determination device shown in FIG. 1. FIG. FIG. 2 is a diagram schematically illustrating an example of a software configuration of a learning device. FIG. 2 is a diagram schematically showing an example of a learning model configured by a neural network. FIG. 3 is a diagram illustrating an example of a method for evaluating performance of a learning model by an evaluation unit. It is a figure which shows an example of a determination parameter. It is a figure explaining the evaluation method of a learning effect. It is a figure which shows an example of the screen for receiving a 1st operation. It is a figure which shows an example of the screen for receiving a 2nd operation. It is a figure which shows an example of history information produced|generated by a recording part. FIG. 7 is a diagram showing another example of history information generated by the recording unit. It is a figure which shows an example of the screen which shows history information. FIG. 7 is a diagram showing another example of a screen showing history information. FIG. 2 is a diagram schematically illustrating an example of a software configuration of a determination device. It is a flowchart which shows an example of the flow of processing of a learning device. FIG. 3 is a diagram showing the evaluation results of the performance of a learning model constructed in the process of repeating machine learning processing according to an experimental example.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the figures are given the same reference numerals and the description thereof will not be repeated.

§1 Application Example An example of a situation where the present invention is applied will be described with reference to FIG. FIG. 1 is a schematic diagram showing the overall configuration of a system according to an embodiment. The system SYS illustrated in FIG. 1 is installed in a manufacturing line or the like, and determines the attributes of a workpiece W by using an image of the workpiece W, which is a product. The attributes of the workpiece W include, for example, the presence or absence of a defect, its type, and the like. As shown in FIG. 1, the system SYS includes a learning device 1 and a determining device 2.

The learning device 1 is a computer configured to generate a learning model 5 by performing machine learning using a learning image group 11 including one or more learning images 101 in which a workpiece W is captured. The learning model 5 receives an input of an image of the workpiece W, and outputs output information for determining the attributes of the workpiece W.

A display device 6 and an input device 7 are connected to the learning device 1 . The display device 6 is typically a liquid crystal display and displays various screens. Input device 7 includes, for example, a keyboard and a mouse. Note that the display device 6 and the input device 7 may be integrated as a touch panel.

The determination device 2 is a computer configured to determine the attributes of the workpiece W using the learning model 5 transferred from the learning device 1. A camera 3 is connected to the determination device 2 . The camera 3 is installed so that the workpiece W is included in its field of view. Thereby, the determination device 2 acquires an observation image in which the workpiece W is captured from the camera 3. The determination device 2 determines the attributes of the workpiece W using the output information of the learning model 5 when the acquired observation image is input.

For example, the determination device 2 determines whether the workpiece W includes a defect. The defects may be, for example, scratches, dirt, cracks, dents, burrs, color unevenness, foreign matter contamination, and the like.

The learning device 1 and the determining device 2 illustrated in FIG. 1 are connected to each other via a network. The type of network may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, and the like.

The method of exchanging data between the learning device 1 and the determination device 2 does not need to be limited to this example, and may be selected as appropriate depending on the embodiment. For example, data may be exchanged between the learning device 1 and the determining device 2 using a storage medium. Furthermore, in this embodiment, the learning device 1 and the determining device 2 are separate computers. However, the configuration of the system SYS does not need to be limited to such an example, and may be determined as appropriate depending on the embodiment. For example, the learning device 1 and the determining device 2 may be an integrated computer. Further, for example, at least one of the learning device 1 and the determining device 2 may be configured by a plurality of computers.

As shown in FIG. 1, the learning device 1 includes a learning section 13. The learning unit 13 generates a learning model 5 used to determine the attributes of the workpiece W by performing machine learning using the learning image group 11. The learning image group 11 includes one or more learning images 101 in which a good workpiece W is captured.

At the start-up stage of the system SYS, the learning image group 11 includes only one or more learning images 101 prepared in advance. Usually, the number of learning images 101 prepared in advance is insufficient, and the performance of the learning model 5 at the start-up stage is not sufficient. Therefore, operations related to additional learning are repeatedly performed, and the performance of the learning model 5 is improved. However, as described above, increasing the number of operations related to additional learning does not necessarily improve the performance of the learning model 5. Therefore, the user cannot easily grasp at which timing the performance of the learning model 5 is highest among the learning models 5 corresponding to each of a plurality of operations performed in the past. As a result, a problem arises in that it takes time and effort to adjust the learning model 5.

The learning device 1 of the system SYS according to the present embodiment further includes an evaluation section 14, a reception section 15, a recording section 16, and an output section 17 in order to solve such problems.

The evaluation unit 14 evaluates the performance of the learning model 5 by inputting one or more evaluation images 102 included in the evaluation image group 12 to the learning model 5. Each evaluation image 102 shows a workpiece W that is a good product or a defective product. The evaluation unit 14 evaluates the performance of the generated learning model 5 every time the learning unit 13 generates the learning model 5.

The reception unit 15 receives a first operation for updating the learning image group 11. The first operation is, for example, an operation in which the determination device 2 adds the observed image acquired from the camera 3 to the learning image group 11 as the learning image 101. The reception unit 15 updates the learning image group 11 in response to the first operation.

When the learning image group 11 is updated by the reception unit 15, the learning unit 13 generates the learning model 5 using the updated learning image group 11. Furthermore, the evaluation unit 14 evaluates the performance of the generated learning model 5.

For each first operation, the recording unit 16 records the content of the first operation in association with the evaluation result for the learning model 5 generated using the updated learning image group 11.

The output unit 17 outputs the history information recorded by the recording unit 16. For example, the output unit 17 displays history information on the display device 6.

By checking the history information recorded by the recording unit 16, the user can determine at which timing the performance of the learning model 5 is highest among the learning models 5 corresponding to each of the plurality of first operations performed in the past. Easy to understand. Thereby, the user can adjust the learning model 5 as appropriate while checking the performance of the learning model 5 for each first operation. Typically, the user sets the learning model 5 with the highest performance in the determination device 2. In this way, according to the present embodiment, the effort required to adjust the learning model 5 can be reduced.

§2 Specific example <A. Example of hardware configuration of learning device>
The learning device 1 is typically a computer having a general-purpose architecture, and executes various processes according to the present embodiment by executing programs (instruction codes) installed in advance. Such programs are typically distributed in a state stored in various recording media or installed in the learning device 1 via a network or the like.

When using such a general-purpose computer, in addition to applications for executing various processes related to this embodiment, an OS (Operating System) for executing basic computer processes must be installed. may have been done. In this case, the program according to the present embodiment may call necessary modules at a predetermined timing in a predetermined sequence among the program modules provided as part of the OS to execute processing. good. That is, the program itself according to this embodiment does not include the above-mentioned modules, and the processing may be executed in cooperation with the OS. The program according to this embodiment may be in a form that does not include some of these modules.

Furthermore, the program according to this embodiment may be provided by being incorporated into a part of another program. Even in that case, the program itself does not include modules included in other programs to be combined as described above, and processing is executed in cooperation with the other programs. That is, the program according to this embodiment may be incorporated into such another program. Note that part or all of the functions provided by executing the program may be implemented as a dedicated hardware circuit.

FIG. 2 is a schematic diagram showing an example of the hardware configuration of the learning device shown in FIG. As shown in FIG. 2, the learning device 1 includes a CPU (Central Processing Unit) 110, a RAM (Random Access Memory) 111, a ROM (Read Only Memory) 112, a communication interface 113, an input interface 114, It includes a display controller 115, a drive 116, and a storage unit 120. These units are connected to each other via a bus so that they can communicate data.

The CPU 110 implements various calculations by loading programs (codes) installed in the storage unit 120 into the RAM 111 and executing them in a predetermined order. RAM 111 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

The communication interface 113 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 113, the learning device 1 can perform data communication with another information processing device (for example, the determination device 2).

Input interface 114 mediates data transmission between CPU 110 and input device 7 . That is, the input interface 114 receives input information input by the user into the input device 7.

The display controller 115 is connected to the display device 6 and controls the screen of the display device 6 so as to notify the user of processing results in the CPU 110 and the like.

The storage unit 120 includes, for example, a hard disk drive, a solid state drive, or the like. The storage unit 120 stores a learning program 122 and an evaluation program 124. The learning program 122 is a program for causing the learning device 1 to execute machine learning processing for generating the learning model 5. The evaluation program 124 is a program for causing the learning device 1 to evaluate the performance of the learning model 5 and record the evaluation results. Each of the learning program 122 and the evaluation program 124 includes a series of information processing instructions.

The storage unit 120 stores one or more learning images 101 and one or more evaluation images 102. The storage unit 120 illustrated in FIG. 2 stores a plurality of learning images 101 and a plurality of evaluation images 102. The learning image 101 is used for machine learning to generate the learning model 5. The evaluation image 102 is used to evaluate the performance of the learning model 5.

The storage unit 120 stores model data 123 that defines a learning model generated by executing the learning program 122. Further, the storage unit 120 stores history information 125 indicating the history of evaluation results generated by executing the evaluation program 124.

The drive 116 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading a program stored in the storage medium 130. The type of drive 116 may be selected as appropriate depending on the type of storage medium 130. At least one of the learning program 122 and the evaluation program 124 may be stored in this storage medium 130.

The storage medium 130 stores information such as a recorded program by electrical, magnetic, optical, mechanical, or chemical action so that a computer, other device, machine, etc. can read the recorded program information. It is a medium for The learning device 1 may acquire at least one of the learning program 122 and the evaluation program 124 from this storage medium 130.

Here, in FIG. 2, a disk-type storage medium such as a CD or a DVD is illustrated as an example of the storage medium 130. However, the type of storage medium 130 is not limited to the disk type, and may be other than the disk type. An example of a storage medium other than a disk type is a semiconductor memory such as a flash memory.

Note that regarding the specific hardware configuration of the learning device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the learning device 1 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA (field-programmable gate array), DSP (digital signal processor), or the like. At least one of the communication interface 113 and the drive 116 may be omitted. The learning device 1 may include, for example, a controller that is connected to an output device such as a speaker other than the display device 6 and controls the output device. The learning device 1 may be composed of multiple computers. In this case, the hardware configurations of the computers may or may not match. Further, the learning device 1 may be an information processing device designed exclusively for the provided service, or may be a general-purpose server device, a general-purpose PC (Personal Computer), or the like.

<B. Hardware configuration example of determination device>
Next, an example of the hardware configuration of the determination device 2 according to this embodiment will be described using FIG. 3. FIG. 3 is a schematic diagram showing an example of the hardware configuration of the determination device shown in FIG.

As shown in FIG. 3, the determination device 2 includes a CPU 210, a RAM 211, a ROM 212, a communication interface 213, an external interface 214, an input interface 215, a display controller 216, a drive 217, and a storage unit 220. include. These units are connected to each other via a bus so that they can communicate data.

The CPU 210 implements various calculations by expanding programs (codes) installed in the storage unit 220 into the RAM 211 and executing them in a predetermined order. RAM 211 is typically a volatile storage device such as DRAM.

The communication interface 213 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The determination device 2 can perform data communication with another information processing device (for example, the learning device 1) by using the communication interface 213.

The external interface 214 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of external interfaces 214 may be selected as appropriate depending on the type and number of external devices to be connected. In this embodiment, the determination device 2 is connected to the camera 3 via an external interface 214.

The camera 3 is used to obtain an observation image 201 of the workpiece W to be inspected. The type and location of the camera 3 are not particularly limited and may be determined as appropriate depending on the embodiment. The camera 3 may be, for example, a general digital camera, a depth camera, an infrared camera, or the like. Further, the camera 3 may be appropriately placed so as to be able to observe the workpiece W being conveyed through the production line. The camera 3 may be placed, for example, near a production line that transports the work W. Note that when the camera 3 includes a communication interface, the determination device 2 may be connected to the camera 3 via the communication interface 213 instead of the external interface 214.

Input interface 215 mediates data transmission between CPU 210 and input device 8. That is, the input interface 215 receives input information input by the user into the input device 8.

The display controller 216 is connected to the display device 9 and controls the screen of the display device 9 so as to notify the user of processing results in the CPU 210 and the like.

The input device 8 is, for example, a device for performing input such as a mouse or a keyboard. The display device 9 is an example of an output device, and is, for example, a display. An operator can operate the determination device 2 via the input device 8 and the display device 9. The input device 8 and the display device 9 may be replaced with touch panel displays.

The storage unit 220 is configured with, for example, a hard disk drive, a solid state drive, or the like. The storage unit 220 stores various information such as a determination program 221 and model data 123.

The determination program 221 is a program for causing the determination device 2 to perform information processing for determining the attributes of the work W reflected in the observation image 201 using the learning model 5 trained by the learning device 1. In the present embodiment, the information processing for determining the attributes of the workpiece W is information processing for determining the quality of the workpiece W. The determination program 221 includes a series of instructions for the information processing.

The drive 217 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 230. At least one of the determination program 221 and the model data 123 may be stored in the storage medium 230. Further, the determination device 2 may acquire at least one of the determination program 221 and the model data 123 from the storage medium 230. The type of storage medium 230 may be a disk type or a type other than a disk type.

Note that regarding the specific hardware configuration of the determination device 2, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the determination device 2 may include multiple hardware processors. The hardware processor may be comprised of a microprocessor, FPGA, DSP, etc. The storage unit 220 may include a RAM 211 and a ROM 212. At least one of the communication interface 213, input interface 215, display controller 216, drive 217, and external interface 214 may be omitted. The determination device 2 may include an output device other than the display device 9, such as a speaker. The determination device 2 may be composed of multiple computers. In this case, the hardware configurations of the computers may or may not match. Further, the determination device 2 may be an information processing device designed exclusively for the provided service, or may be a general-purpose server device, a general-purpose PC, or the like.

<C. Example of software configuration of learning device>
Next, an example of the software configuration of the learning device 1 according to this embodiment will be described using FIG. 4. FIG. 4 is a diagram schematically illustrating an example of the software configuration of the learning device.

As shown in FIG. 4, the learning device 1 includes a learning section 13, an evaluation section 14, a reception section 15, a recording section 16, an output section 17, and a setting section 18 as software modules. The learning unit 13 is realized by the CPU 110 interpreting and executing instructions included in the learning program 122. The evaluation section 14, reception section 15, recording section 16, output section 17, and setting section 18 are realized by the CPU 110 interpreting and executing instructions included in the evaluation program 124. The specific processing of each software module will be described below.

(C-1. Learning Department)
The learning unit 13 generates the learning model 5 by performing machine learning using one or more learning images 101 included in the learning image group 11. Each learning image 101 shows a good workpiece W. When an image is input, the learning model 5 exemplified in this embodiment converts the input image (hereinafter referred to as "input image") into a feature amount, and uses the feature amount obtained by the conversion. , is a generation model configured to generate an image obtained by restoring an input image (hereinafter referred to as a "restored image"). By machine learning, when one or more learning images 101 are input, the learning unit 13 causes the learning model 5 to generate a restored image that matches each of the input one or more learning images 101. train.

Specifically, the learning model 5 is constituted by a module including calculation parameters used for calculation processing to convert an input image into a feature amount and generate a restored image from the feature amount obtained by the conversion. In machine learning, when a learning image 101 is given, the values of calculation parameters are adjusted so as to generate a restored image that matches the learning image 101.

The learning unit 13 generates the learning model 5 using preset parameters (hereinafter referred to as "model generation parameters"). The model generation parameters are appropriately set according to the structure of the learning model 5.

The configuration of the learning model 5 does not need to be particularly limited, and may be appropriately selected depending on the embodiment. For example, the learning model 5 is configured by a neural network, eigenvectors derived by principal component analysis, and the like. Examples of neural networks include the autoencoder disclosed in Non-Patent Document 1, GAN (Generative Adversarial Networks) disclosed in Non-Patent Document 2, and the like. Examples of the eigenvector include the eigenvector of a subspace disclosed in Non-Patent Document 3.

FIG. 5 is a diagram schematically showing an example of a learning model configured by a neural network. An autoencoder is a typical example of a neural network. An autoencoder is a neural network intended for dimensionality reduction. FIG. 5 shows a learning model 5 composed of an autoencoder.

The neural network configuring the learning model 5 illustrated in FIG. 5 includes an input layer 51, an intermediate (hidden) layer 52, and an output layer 53. However, the structure of the neural network does not need to be limited to such an example, and may be determined as appropriate depending on the embodiment. For example, the number of intermediate layers is not limited to one, and may be two or more.

The number of neurons (nodes) included in the input layer 51 corresponds to the number of pixels of the input image. The number of neurons included in the intermediate layer 52 corresponds to the number of dimensions of the feature amount. The number of neurons included in the output layer 53 corresponds to the number of pixels of the restored image. The number of neurons included in the intermediate layer 52 is set to be smaller than the number of neurons included in each of the input layer 51 and the output layer 53. Accordingly, the number of dimensions of the feature amount is set smaller than the number of dimensions of the input image and restored image.

In a neural network, neurons in adjacent layers are appropriately connected, and a weight (connection weight) is set for each connection. In the learning model 5 illustrated in FIG. 5, each neuron is connected to all neurons in adjacent layers. However, the connection of neurons does not need to be limited to this example, and may be set as appropriate depending on the embodiment. A threshold value is set for each neuron, and basically, the output of each neuron is determined depending on whether the sum of the products of each input and each weight exceeds the threshold value. The weight of the connection between each neuron included in each layer 51 to 53 and the threshold value of each neuron are examples of calculation parameters of the learning model.

The learning unit 13 prepares a neural network that constitutes the learning model 5. When generating the learning model 5 composed of a neural network, the model creation parameters include the number of layers, the number of neurons included in each layer, the connection relationship between neurons in adjacent layers, and the initial weight of the connection between each neuron. value, the initial value of the threshold value of each neuron, etc. are set. Model creation parameters may be provided by a template or may be provided by user input.

The learning unit 13 uses each learning image 101 to execute a learning process for each neural network. This learning process may use a batch gradient descent method, a stochastic gradient descent method, a mini-batch gradient descent method, or the like. For example, in the first step, the learning unit 13 inputs each learning image 101 to the input layer 51 of each neural network, and executes arithmetic processing of each neural network. That is, the learning unit 13 inputs each learning image 101 to the input layer 51 of each neural network, and determines the firing of each neuron included in each layer 51 to 53 in order from the input side. Through this calculation process, the learning unit 13 converts each learning image 101 into a feature amount, and outputs an output value corresponding to the result of generating an image in which each learning image 101 is restored from the feature amount obtained by the conversion. Obtained from layer 53. In the process of this calculation process, the output obtained from the intermediate layer 52 corresponds to the result of converting the input image into a feature amount.

In the second step, the learning unit 13 calculates the error between the output value obtained from the output layer 53 and each learning image 101 based on the loss function. The type of loss function does not need to be particularly limited, and may be selected as appropriate depending on the embodiment. A known loss function may be used as the loss function.

In the third step, the learning unit 13 uses the error of the calculated output value by the error back propagation method to determine the calculation parameters of each neural network, that is, between each neuron in each layer 51 to 53. Calculate the connection weight and the error of each neuron's threshold.

In the fourth step, the learning unit 13 updates the weight of the connection between each neuron in each layer 51 to 53 and the threshold value of each neuron based on each calculated error.

By repeating the first to fourth steps described above, the learning unit 13 performs calculations so that the sum of errors between the output value (restored image) output from the output layer 53 and each learning image 101 becomes small. Adjust parameter values. For example, the learning unit 13 repeats the adjustment of the calculation parameters in the first to fourth steps described above until the sum of the errors between the output value output from the output layer 53 and each learning image 101 becomes equal to or less than the threshold value. It's okay. The threshold value may be set as appropriate depending on the embodiment.

As a result, the learning unit 13 uses, as the learning model 5, a neural network that is trained to output a restored image matching each learning image 101 from the output layer 53 when each learning image 101 is input to the input layer 51. Can be built. In the neural network, each input learning image 101 is converted into a feature amount during the calculation process from the input layer 51 to the intermediate layer 52. Then, a restored image is generated from the obtained feature amounts through the calculation process from the intermediate layer 52 to the output layer 53.

Each learning image 101 shows a good workpiece W. Therefore, the learning model 5 generates a restored image that has high reproducibility for the features that appear in the training image 101 and has low reproducibility for defects that are unlikely (for example, have no possibility) to appear in the training image 101. have acquired the ability to

The learning unit 13 generates information that can reproduce the values of calculation parameters in the finally constructed trained neural network (learning model 5) as model data 123, and stores it in the storage unit 120. The learning unit 13 transfers the model data 123 to the determination device 2. The determination device 2 acquires the model data 123 by accepting this transfer. Alternatively, the determination device 2 may obtain the model data 123 by accessing the learning device 1 or the data server via the network using the communication interface 213. Further, for example, the determination device 2 may acquire the model data 123 via the storage medium 230.

Note that the type of neural network constituting the learning model 5 is not limited to such a fully connected neural network with a multilayer structure. The learning model 5 may use a convolutional neural network including a convolution layer and a pooling layer.

(C-2. Evaluation Department)
The evaluation unit 14 evaluates the performance of the learning model 5 by inputting one or more evaluation images 102 included in the evaluation image group 12 to the learning model 5. Hereinafter, a specific example of processing by the evaluation unit 14 will be described with reference to FIGS. 6 and 7.

FIG. 6 is a diagram illustrating an example of a method for evaluating the performance of a learning model by the evaluation unit. The evaluation image group 12 includes one or more first evaluation images 103 and one or more second evaluation images 104. The evaluation image group 12 illustrated in FIG. 6 includes a plurality of first evaluation images 103 and a plurality of second evaluation images 104. Each first evaluation image 103 shows a workpiece W that is a non-defective product. Each second evaluation image 104 shows a workpiece W that is a defective product. For example, the second evaluation image 104 shows a workpiece W including a defect L.

The evaluation unit 14 inputs the plurality of first evaluation images 103 into the learning model 5 to obtain a plurality of first restored images 105 that are each restored from the plurality of first evaluation images 103 . The first restored image 105 is output information of the learning model 5. The evaluation unit 14 determines the quality of the workpiece W shown in the corresponding first evaluation image 103 depending on whether each first restored image 105 satisfies the determination conditions.

Specifically, the evaluation unit 14 generates a difference image 107 showing the difference between each first evaluation image 103 and each first restored image 105, and uses the difference image 107 and the determination parameter 56 to determine each first evaluation image 103 and each first restored image 105. It is determined whether the defect L is shown in the first evaluation image 103. The determination parameter 56 defines the determination condition. When the defect L is shown in the first evaluation image 103, the evaluation unit 14 determines that the work W shown in the first evaluation image 103 is a defective product.

Similarly, the evaluation unit 14 inputs the plurality of second evaluation images 104 to the learning model 5, thereby obtaining a plurality of second restored images 106 that are each restored from the plurality of second evaluation images 104. The second restored image 106 is output information of the learning model 5. The evaluation unit 14 determines the quality of the workpiece W shown in the corresponding second evaluation image 104 depending on whether each second restored image 106 satisfies the determination conditions.

Specifically, the evaluation unit 14 generates a difference image 108 showing the difference between each second evaluation image 104 and each second restored image 106, and uses the difference image 108 and the determination parameter 56 that defines the determination condition. Then, it is determined whether the defect L is included in each second evaluation image 104. If the defect L is shown in the second evaluation image 104, the evaluation unit 14 determines that the workpiece W shown in the second evaluation image 104 is defective.

As described above, the evaluation unit 14 is provided with the determination parameters 56 that define the determination conditions for determining whether the defect L is included in each image (103, 104). For example, the value of the determination parameter 56 may be specified by operator input, or may be given as a set value within the program. In this embodiment, a plurality of determination parameter candidates 55 are given, and one of the plurality of determination parameter candidates 55 is determined as the determination parameter 56. Each determination parameter candidate 55 indicates a value of a determination parameter candidate. The value of each determination parameter candidate 55 may be given as appropriate.

The evaluation unit 14 uses each determination parameter candidate 55 to determine whether the defect L is depicted in each first evaluation image 103 and whether or not the defect L is depicted in each second evaluation image 104. Execute the judgment.

Based on the results of each determination, the evaluation unit 14 determines which of the plurality of first evaluation images 103 is the first evaluation image 103 in which the workpiece W is determined to be defective (that is, it is determined that the defect L is captured). A first evaluation value is calculated according to the first ratio of the first evaluation image 103). The first evaluation value is, for example, a value expressed as a percentage of the first ratio (hereinafter referred to as "overlooking rate"). Note that the first evaluation value is not limited to the overwatch rate, and may be a value obtained by subtracting the overwatch rate from 100.

Furthermore, based on the results of each determination, the evaluation unit 14 determines, among the plurality of second evaluation images 104, that the second evaluation image 104 in which the workpiece W is determined to be non-defective (that is, the defect L is not included in the second evaluation image 104) A second evaluation value is calculated according to the second ratio of the determined second evaluation image 104). The second evaluation value is, for example, a value representing the second ratio as a percentage (hereinafter referred to as "miss rate"). Note that the second evaluation value is not limited to the missed rate, and may be a value obtained by subtracting the missed rate from 100.

In the production line, it is important to avoid shipping defective products as non-defective products as much as possible. Therefore, the evaluation unit 14 selects one judgment parameter candidate whose second ratio is equal to or less than a predetermined threshold (hereinafter referred to as "acceptable missed rate") among the plurality of judgment parameter candidates 55 as the judgment parameter 56. decide. The allowable missed rate is set in advance, for example, by user input. The allowable missed rate is, for example, 0. If there are two or more determination parameter candidates whose second ratio is less than or equal to the allowable missed rate, the evaluation unit 14 selects one determination parameter candidate whose first ratio is the smallest among the two or more determination parameter candidates as the determination parameter 56. Determine as. If there is no determination parameter candidate whose second ratio is less than or equal to the allowable missed rate, the evaluation unit 14 determines one determination parameter candidate whose second ratio is the smallest among the plurality of determination parameter candidates 55 as the determination parameter 56. Thereby, the evaluation unit 14 automatically optimizes the determination conditions for determining the quality of the workpiece W.

The evaluation unit 14 outputs the first evaluation value and the second evaluation value calculated in the determination using the determined determination parameter 56 as an evaluation result of the performance of the learning model 5.

FIG. 7 is a diagram showing an example of determination parameters. In FIG. 7, an image I10 corresponds to each of the first evaluation image 103 and the second evaluation image 104 shown in FIG. The restored image I11 corresponds to each of the first restored image 105 and the second restored image 106 shown in FIG. 7.

The evaluation unit 14 generates a difference image I12 by calculating the difference between the image I10 and the restored image I11. The difference image I12 corresponds to each of the difference images 107 and 108 shown in FIG. The pixel value that is more different between the image I10 and the restored image I11 becomes larger in the difference image I12. On the other hand, the less different a pixel is between the image I10 and the restored image I11, the smaller the pixel value in the difference image I12. In this embodiment, for convenience of explanation, the larger the difference between a pixel between the image I10 and the restored image I11, the larger the actual pixel value of the corresponding pixel in the difference image I12, and the smaller the difference is, the larger the actual pixel value of the corresponding pixel in the difference image I12. Assume that the actual pixel value of the corresponding pixel within is also smaller. However, "the pixel value is large" and "the pixel value is small" each indicate the relationship between the difference between the image I10 and the restored image I11, and the actual pixel value of the pixel in the difference image I12. It doesn't have to be compatible. For example, the difference image I12 may be calculated such that the larger the difference between pixels, the smaller the actual pixel value of the corresponding pixel, and the smaller the difference between the pixels, the larger the actual pixel value of the corresponding pixel.

In the example of FIG. 7, the pixels that have a difference between the image I10 and the restored image I11 are white in the difference image I12, and the pixels that are not so different are black. For example, when the value of each pixel is expressed in 256 gradations, the maximum value of the pixel value of the pixels of the difference image I12 may be "255", and the minimum value may be "0". In this case, the larger the pixel value of the pixel of the difference image I12, the more difference occurs between the image I10 and the restored image I11, and the smaller the pixel value of the pixel of the difference image I12, the more the difference between the image I10 and the restored image This shows that there is no difference between the image I11 and the image I11. However, the relationship between the difference that occurs between the image I10 and the restored image I11 and the pixel values of the difference image I12 is not limited to this example. For example, the relationship between the degree of difference between the image I10 and the restored image I11 and the pixel value of the difference image I12 may be the opposite.

If a feature (such as a defect L) that is not included or is unlikely to be included in the learning image 101 used for machine learning of the learning model 5 is included in the image I10, the reproducibility of that feature is low in the restored image I11. . Therefore, a relatively large difference may occur between the image I10 and the restored image I11. However, the cause of the difference between the image I10 and the restored image I11 is not limited to the fact that such a defect L appears in the image I10. Another cause of a relatively large difference between the image I10 and the restored image I11 is that, for example, the appearance of the workpiece W in the input image may be different from the appearance of the workpiece W in the learning image 101 used for machine learning. There are some differences between the two. In this case, the appearance of the workpiece W reflected in the image I10 may not be completely reproduced in the restored image I11, which may cause a relatively large difference between the image I10 and the restored image I11.

Further, for example, the learning model 5 compresses the input image into a low-dimensional feature amount. At this time, information of the input image may be partially lost. Therefore, some error may occur between the input image and the restored image. This restoration noise can be an example of the cause of the difference between the image I10 and the restored image I11. Therefore, in addition to the difference caused by the defect L, differences based on a plurality of factors may appear in the difference image I12. However, the difference due to this noise is of a lower magnitude than the difference due to the above two factors. Therefore, these differences can be distinguished based on the pixel values of the difference image I12.

The evaluation unit 14 binarizes each pixel of the difference image I12 using the threshold value 57. For example, the evaluation unit 14 converts the pixel value of a pixel whose pixel value is equal to or greater than the threshold value 57 to "255", and converts the pixel value of a pixel whose pixel value is less than the threshold value 57 to "0". “More than” may be replaced with “more than” and “less than” may be replaced with “less than”. The same applies to the following description. Thereby, the evaluation unit 14 can generate the binarized image I13. By appropriately setting the threshold value 57, it is possible to obtain a binarized image I13 in which relatively low-level differences such as those caused by the noise are excluded from the original difference image I12.

Differences due to the defect L and differences due to insufficient learning may appear in the binarized image I13. Among the causes of these differences, the defect L may have attributes related to shape, such as area, width, height, circumference, aspect ratio, and circularity. That is, when a defect L exists in the workpiece W shown in the image I10, the defect L is located in an area where white "255" pixels (hereinafter also referred to as white pixels) are gathered at the corresponding position in the binarized image I13. A region having shape-related attributes equivalent to L appears. Therefore, by setting the threshold value 58 for the attribute related to the shape, it is possible to determine whether the defect L is included in the binarized image I13.

As an example of the process, the evaluation unit 14 identifies a region of continuous white pixels in the binarized image I13 as one region, and determines whether each region of white pixels satisfies the threshold value 58. Then, the evaluation unit 14 leaves the area that satisfies the threshold value 58 as it is, and converts the pixel values of pixels in the area that does not satisfy the threshold value 58 to "0". For example, when the threshold value 58 is set for area, the evaluation unit 14 determines whether the area of each region of white pixels is equal to or larger than the threshold value 58. Then, the evaluation unit 14 converts the pixel values of pixels within the area whose area is less than the threshold value 58 to "0". Thereby, the evaluation unit 14 can generate the detected image I14. By appropriately setting the threshold value 58, it is possible to obtain a detected image I14 in which white areas that do not satisfy the attributes of the defect L are excluded from the binarized image I13.

The evaluation unit 14 determines whether the defect L is included in the image I10, depending on whether a white pixel area exists in the detected image I14. Specifically, when a region of white pixels exists in the detected image I14, the evaluation unit 14 determines that the defect L is reflected in the corresponding region of the image I10. On the other hand, if there is no area of white pixels in the detected image I14, the evaluation unit 14 determines that the defect L is not included in the image I10.

In the processing procedure regarding the detection of the defect L described above, the threshold values 57 and 58 are examples of determination parameters. That is, in this embodiment, each determination parameter candidate 55 is configured by a combination of threshold values 57 and 58. However, the method for detecting the defect L and the determination parameters are not limited to these examples, and may be appropriately selected depending on the embodiment.

(C-3. Reception Department)
As described above, at the start-up stage of the system SYS, the number of learning images 101 included in the learning image group 11 is insufficient, and the performance of the learning model 5 at the start-up stage is not sufficient.

For example, individual differences may occur in the work W. Furthermore, for example, photographing conditions such as the timing of photographing may change (for example, the position of the workpiece W may shift or be tilted). Depending on these factors, the appearance of the work W appearing in the obtained image may vary. At the start-up stage of the system SYS, it is difficult to determine whether the learning model 5 has acquired the ability to restore the image of the workpiece W in a range other than the range appearing in the learning image 101 included in the learning image group 11. It is unknown.

Therefore, in the system SYS, operations related to additional learning are repeatedly performed to improve the performance of the learning model 5.

The reception unit 15 receives operations related to additional learning. The receiving unit 15 illustrated in FIG. 6 receives a first operation for updating the learning image group 11 and a second operation for updating the evaluation image group 12. The reception unit 15 updates the learning image group 11 according to the received first operation. The reception unit 15 updates the evaluation image group 12 according to the received second operation.

The reception unit 15 may read images to be added to the learning image group 11 or the evaluation image group 12 from the storage medium 130. Alternatively, the receiving unit 15 may acquire images to be added to the learning image group 11 or the evaluation image group 12 from an external information processing device. Alternatively, the reception unit 15 acquires a plurality of observation images 201 from the determination device 2, and acquires a part of the plurality of observation images 201 as an image to be added to the learning image group 11 or the evaluation image group 12. Good too. An example of adding a part of the observed image 201 to the learning image group 11 or the evaluation image group 12 will be described below.

The plurality of observed images 201 acquired from the determination device 2 include images that have a high learning effect and images that do not. It is preferable to add images with a high learning effect to the learning image group 11 as the learning images 101. Therefore, in order to support the first operation, the reception unit 15 evaluates the learning effect for each of the plurality of observation images 201, and notifies the user of the evaluation result.

FIG. 8 is a diagram illustrating a method for evaluating learning effects. The receiving unit 15 obtains a plurality of restored observation images 203 obtained by restoring each of the plurality of observation images 201 by giving each of the plurality of acquired observation images 201 to the learning model 5. Then, the reception unit 15 evaluates the learning effect according to the difference between each observation image 201 and each restored observation image 203. Specifically, the receiving unit 15 calculates an evaluation value (hereinafter referred to as a "learning effect evaluation value") for the degree of restoration of each observed image 201 in each restored observed image 203.

The learning effect evaluation value is calculated as follows, for example. The reception unit 15 generates a difference image between each observation image 201 and each restored observation image 203. A pixel with a large pixel value in the difference image corresponds to a pixel with a large difference between each observation image 201 and each restored observation image 203. A pixel with a small pixel value in the difference image corresponds to a pixel with a small difference between each observation image 201 and each restored observation image 203. The receiving unit 15 calculates the frequency for each pixel value according to the pixel value of each pixel in the difference image. "Frequency" indicates the number of pixels with the same pixel value. Subsequently, the reception unit 15 extracts pixels from the highest rank to a predetermined rank when arranged in descending order according to pixel values in the difference image. The predetermined ranking may be specified by an operator's input, a set value in a program, or the like. Then, the reception unit 15 calculates the average value or total value of the pixel values of the extracted pixels as a learning effect evaluation value. As a result, the larger the calculated learning effect evaluation value, the larger the difference between the observed image 201 and the restored observed image 203, in other words, the lower the degree of restoration of the observed image 201 in the restored observed image 203. .

However, the method of calculating the learning effect evaluation value is not limited to this example. The relationship between the difference between the observation image 201 and the restored observation image 203 and the learning effect evaluation value is such that the learning effect evaluation value indicates that the degree of restoration is low as the difference is large, and the difference is It may be determined as appropriate so that the smaller the value, the higher the degree of restoration. For example, contrary to the calculation method described above, the learning effect evaluation value may be calculated such that the lower the degree of restoration, the smaller the learning effect evaluation value, and the higher the degree of restoration, the larger the learning effect evaluation value. As an example of the calculation method, the receiving unit 15 may calculate the average value or the reciprocal of the total value of the pixel values of the pixels extracted by the above process as the learning effect evaluation value.

The larger the difference between the observed image 201 and the restored observed image 203, the lower the degree of restoration of the observed image 201 in the restored observed image 203. If the workpiece W shown in the observation image 201 is a good product, this indicates that the ability of the learning model 5 to reproduce the state of the workpiece W shown in the observation image 201 is insufficient. Therefore, the observed image 201 with a large difference from the restored observed image 203, in other words, the observed image 201 with a low degree of restoration, is likely to contribute to appropriate improvement of the ability of the learning model 5 to reproduce the image in which the workpiece W is captured.

On the contrary, the observed image 201 with a high degree of restoration is unlikely to contribute to appropriate improvement of the ability of the learning model 5. In addition, by adding the observed image 201 with a high degree of restoration to the learning image group 11 as the learning image 101, the ability to restore patterns other than the workpiece W may be improved. As a result, the learning model 5 may acquire the ability to faithfully reproduce any pattern.

Taking these points into consideration, the reception unit 15 provides the user with a learning effect evaluation value as an index for selecting an image to be added to the learning image group 11 as the learning image 101 from among the plurality of observed images 201. Notice.

Furthermore, there is a possibility that the plurality of observation images 201 include defective workpieces W. If the observed image 201 showing the defective workpiece W is added to the learning image group 11 as the learning image 101, it becomes impossible to accurately determine the quality of the workpiece W using the output information of the learning model 5. Therefore, the reception unit 15 displays each of the plurality of observation images 201 on the display device 6, and allows the user to select the observation image 201 in which the good workpiece W is captured as the learning image 101.

FIG. 9 is a diagram showing an example of a screen for accepting the first operation. A screen 60 illustrated in FIG. 9 is displayed on the display device 6 by the reception unit 15.

The screen 60 includes a candidate list 61, a learning image list 62, an area 63, an add button 64, a delete button 65, and a learning execution button 66.

The candidate list 61 is a list of a plurality of observation images 201 acquired from the determination device 2. The candidate list 61 indicates a file name and a corresponding learning effect evaluation value for each observed image 201.

The learning image list 62 is a list of the learning images 101 included in the learning image group 11. The learning image list 62 shows the file name of each learning image 101.

In the area 63, one image selected from the candidate list 61 and the learning image list 62 is displayed.

The user checks the learning effect evaluation values in the candidate list 61 and selects the observed image 201 that is evaluated as having a high learning effect. The user looks at the observation image 201 displayed in the area 63 and confirms whether the workpiece W shown in the observation image 201 is a good product. Thereby, the user can select, as the learning image 101, the observation image 201 that shows a good workpiece W and has a high learning effect.

The add button 64 is a button for accepting an operation (one of the first operations) for adding the observed image 201 selected from the candidate list 61 to the learning image group as the learning image 101. When the add button 64 is operated, the reception unit 15 adds the observation image 201 selected from the candidate list 61 to the learning image group as the learning image 101. As a result, the learning image list 62 is also updated.

The delete button 65 is a button for accepting an operation (one of the first operations) for deleting the learning image 101 selected from the learning image list 62 from the learning image group 11. For example, the user selects the unnecessary learning image 101 from the learning image list 62 while looking at the area 63, and operates the delete button 65. When the delete button 65 is operated, the reception unit 15 deletes the learning image 101 selected from the learning image list 62 from the learning image group 11. As a result, the learning image list 62 is also updated.

The learning execution button 66 is a button for instructing execution of machine learning using the updated learning image group 11. When the learning execution button 66 is operated, the learning unit 13 generates the learning model 5 by performing machine learning using the updated learning image group 11. Furthermore, the evaluation unit 14 evaluates the performance of the generated learning model 5. Model data 123 indicating the generated learning model 5 is transferred to the determination device 2. Further, the determination parameter 56 determined by the evaluation unit 14 is also transferred to the determination device 2.

FIG. 10 is a diagram showing an example of a screen for accepting the second operation. A screen 70 illustrated in FIG. 10 is displayed on the display device 6 by the reception unit 15.

The screen 70 includes a candidate list 71, a good product image list 72, a defective product image list 73, an area 74, add buttons 75a and 75b, a move button 76, delete buttons 77a and 77b, and an evaluation execution button 78. including.

The candidate list 71 is a list of a plurality of observation images 201 acquired from the determination device 2, similar to the candidate list 61 shown in FIG. The non-defective image list 72 is a list of the first evaluation images 103 included in the evaluation image group 12. The non-defective image list 72 shows the file name of each first evaluation image 103. The defective product image list 73 is a list of the second evaluation images 104 included in the evaluation image group 12. The defective product image list 73 shows the file name of each second evaluation image 104. In the area 74, one image selected from the candidate list 71, the non-defective image list 72, and the defective image list 73 is displayed.

The add button 75a is a button for accepting an operation (one of the second operations) for adding the observed image 201 selected from the candidate list 71 to the evaluation image group 12 as the first evaluation image 103. The user selects the observation image 201 showing the good workpiece W from the candidate list 71 while looking at the area 74, and operates the add button 75a. When the add button 75a is operated, the reception unit 15 adds the observation image 201 selected from the candidate list 71 to the evaluation image group 12 as the first evaluation image 103. As a result, the non-defective image list 72 is also updated.

The add button 75b is a button for accepting an operation (one of the second operations) for adding the observed image 201 selected from the candidate list 71 to the evaluation image group 12 as the second evaluation image 104. The user selects the observation image 201 in which the defective workpiece W appears from the candidate list 71 while looking at the area 74, and operates the add button 75b. When the add button 75b is operated, the reception unit 15 adds the observation image 201 selected from the candidate list 71 to the evaluation image group 12 as the second evaluation image 104. As a result, the defective product image list 73 is also updated.

The move button 76 is a button for accepting an operation (one of the second operations) of moving one image selected from either the non-defective product image list 72 or the defective product image list 73 to the other. For example, when the user confirms that a defective workpiece W is captured in the first evaluation image 103 included in the non-defective image list 72 while looking at the area 74, the user Select image 103 and operate move button 76. Alternatively, when the user confirms that the non-defective workpiece W is captured in the second evaluation image 104 included in the defective product image list 73 while looking at the area 74, the user selects a second evaluation image in which the non-defective workpiece W is captured. 104 and operate the move button 76.

When the move button 76 is operated with one first evaluation image 103 in the non-defective image list 72 selected, the reception unit 15 transfers the selected first evaluation image 103 to the second evaluation image. Change it to 104. When the move button 76 is operated with one second evaluation image 104 in the defective product image list 73 selected, the reception unit 15 selects the selected second evaluation image 104 as the first evaluation image. Change to image 103. As a result, the non-defective image list 72 and the defective image list 73 are also updated.

The delete button 77a is a button for accepting an operation (one of the second operations) for deleting the first evaluation image 103 selected from the non-defective image list 72 from the evaluation image group 12. When the delete button 77a is operated, the reception unit 15 deletes the first evaluation image 103 selected from the non-defective image list 72 from the evaluation image group 12. As a result, the non-defective image list 72 is also updated.

The delete button 77b is a button for accepting an operation (one of the second operations) for deleting the second evaluation image 104 selected from the defective product image list 73 from the evaluation image group 12. When the delete button 77b is operated, the reception unit 15 deletes the second evaluation image 104 selected from the defective product image list 73 from the evaluation image group 12. As a result, the defective product image list 73 is also updated.

The evaluation execution button 78 is a button for instructing execution of evaluation using the updated evaluation image group 12. When the evaluation execution button 78 is pressed, the evaluation unit 14 re-evaluates the performance of the learning model 5 using the updated evaluation image group. The determination parameters determined by the evaluation unit 14 at this time are transferred to the determination device 2.

(C-4. Recording Department)
Each time the reception unit 15 receives an operation related to machine learning (a first operation and a second operation), the recording unit 16 records the contents of the received operation in association with the results of evaluating the performance of the learning model 5. . The recording unit 16 adds the recording results to the history information 125 stored in the storage unit 120.

FIG. 11 is a diagram illustrating an example of history information generated by the recording unit. As shown in FIG. 11, the history information 125 includes, for each operation, the operation details, the performance evaluation result of the learning model 5, the model data 123 indicating the learning model 5, the number of learning images 101, and the evaluation. This is a table in which the determination parameters 56 determined by the unit 14 are associated with each other. In the history information 125 illustrated in FIG. 11, "overlook rate" and "overlook rate" are recorded as evaluation results. Moreover, the values of threshold values 57 and 58 shown in FIG. 7 are recorded as determination parameters. Note that the history information 125 may include the date and time when the operation was received.

For example, when the receiving unit 15 accepts the first operation, the recording unit 16 records the updated contents of the learning image group 11 and the performance evaluation result of the learning model 5 generated using the updated learning image group 11. , creates a record that associates the model data 123 that defines the learning model 5, the number of learning images 101 included in the updated learning image group 11, and the determination parameter 56 determined by the evaluation unit 14. . The recording unit 16 adds the created record to the end of the history information 125.

When the receiving unit 15 accepts the second operation, the recording unit 16 records the updated contents of the evaluation image group 12, the performance evaluation results using the updated evaluation image group 12 for the current learning model 5, and A record is created in which the model data 123 that defines the current learning model 5, the number of learning images 101 included in the current learning image group 11, and the determination parameter 56 determined by the evaluation unit 14 are associated. The recording unit 16 adds the created record to the end of the history information 125.

FIG. 12 is a diagram showing another example of history information generated by the recording unit. As shown in FIG. 12, the history information 125 includes, for each operation, the operation details, the performance evaluation result of the learning model 5, the list of learning images 101 included in the learning image group 11, and the evaluation images. This is a table in which lists of the first evaluation images 103 and second evaluation images 104 included in the group 12 are associated with the determination parameters 56 determined by the evaluation unit 14. Note that the history information 125 may include the date and time when the operation was received.

For example, when the receiving unit 15 accepts the first operation, the recording unit 16 records the updated contents of the learning image group 11 and the performance evaluation result of the learning model 5 generated using the updated learning image group 11. , a list of learning images 101 included in the updated learning image group 11, a list of each of the first evaluation images 103 and second evaluation images 104 included in the current evaluation image group 12, and the evaluation unit. A record is created in which the determination parameters 56 determined in step 14 are associated with each other. The recording unit 16 adds the created record to the end of the history information 125.

When the receiving unit 15 accepts the second operation, the recording unit 16 records the updated contents of the evaluation image group 12, the performance evaluation results using the updated evaluation image group 12 for the current learning model 5, and A list of learning images 101 included in the current learning image group 11 , a list of each of the first evaluation images 103 and second evaluation images 104 included in the updated evaluation image group 12 , and the evaluation unit 14 A record is created in which the determination parameters 56 determined by the above are associated with each other. The recording unit 16 adds the created record to the end of the history information 125.

In this way, the history of operations accepted by the reception unit 15 is accumulated in the history information 125.

(C-5. Output section)
The output unit 17 outputs the history information 125 recorded by the recording unit 16 in response to user input. Specifically, when an instruction to output the history information 125 is input to the input device 7, the output unit 17 causes the display device 6 to display a screen showing the history information 125.

FIG. 13 is a diagram showing an example of a screen showing history information. The screen 80 illustrated in FIG. 13 is created, for example, based on the history information 125 illustrated in FIG. 12. Screen 80 is displayed on display device 6 by output unit 17 .

The screen 80 includes an operation history table 81 , a learning image list 83 , a defective product image list 84 , a non-defective product image list 85 , an area 86 , and a display field 87 .

The operation history table 81 is a table showing, for each operation, the operation details, the performance evaluation result of the learning model 5, and the determination parameter 56 determined by the evaluation unit 14. The operation history table 81 is created using, for example, the history information 125 shown in FIG. 12.

In the display field 87, the allowable missed rate set for determining the determination parameter 56 is displayed.

On the screen 80 illustrated in FIG. 13, "overlook rate" and "overlook rate" are displayed as the evaluation results of the performance of the learning model 5. It is preferable that the output unit 17 makes the display format of records whose "miss rate" exceeds the allowable missed rate different from other records in the operation history table 81. Thereby, it is possible to call attention to the user.

The operation history table 81 displays a cursor 82 for selecting one operation (hereinafter referred to as "target operation"). The cursor 82 moves according to input to the input device 7.

The learning image list 83 shows a list of the learning images 101 included in the learning image group 11 immediately after the target operation selected by the cursor 82 is performed. The learning image list 83 is created based on the "learning image group" field of the record corresponding to the target operation in the history information 125 shown in FIG. 12.

The defective product image list 84 shows a list of the second evaluation images 104 included in the evaluation image group 12 immediately after the target operation selected by the cursor 82 is performed. The defective product image list 84 is created based on the "second evaluation image" field of the record corresponding to the target operation in the history information 125 shown in FIG. 12.

The non-defective image list 85 shows a list of the first evaluation images 103 included in the evaluation image group 12 immediately after the target operation selected by the cursor 82 is performed. The non-defective image list 85 is created based on the "first evaluation image" field of the record corresponding to the target operation in the history information 125 shown in FIG.

The defective product image list 84 and the non-defective product image list 85 display, for each image, the determination result of the quality of the workpiece W shown in the image. In the defective product image list 84, "OK" indicates that the workpiece W has been determined to be a defective product, and "missed" indicates that the workpiece W has been determined to be a non-defective product. In the non-defective image list 85, "OK" indicates that the workpiece W has been determined to be a non-defective product, and "overlooked" indicates that the workpiece W has been determined to be a defective product.

In the area 86, one image selected from the learning image list 83, the defective product image list 84, and the non-defective product image list 85 is displayed.

FIG. 14 is a diagram showing another example of a screen showing history information. The screen 90 illustrated in FIG. 14 is created based on the history information 125 illustrated in FIG. 11 or 12, for example. Screen 90 is displayed on display device 6 by output unit 17 . Note that the screen 80 shown in FIG. 13 and the screen 90 shown in FIG. 14 may be switched depending on the input to the input device 7.

Screen 90 includes an operation history graph 91 and display fields 93-95. The operation history graph 91 is a graph showing changes in the performance evaluation result of the learning model 5 and the number of learning images 101 included in the learning image group 11.

The operation history graph 91 displays a cursor 92 for selecting one operation (hereinafter referred to as "target operation"). The cursor 92 moves according to input to the input device 7.

The display field 93 displays the contents of the target operation selected by the cursor 92. The content displayed in the display column 93 corresponds to the "operation content" field of the record corresponding to the target operation in the history information 125 shown in FIG. 12 or 13.

The display field 94 displays the performance evaluation result of the learning model 5 and the number of learning images 101 included in the learning image group 11 immediately after the target operation selected by the cursor 92 is executed. . The evaluation result displayed in the display field 94 corresponds to the "performance evaluation result" field of the record corresponding to the target operation in the history information 125 shown in FIG. 12 or 13. The number of learning images 101 displayed in the display field 94 corresponds to the "number of learning images" field of the record corresponding to the target operation in the history information 125 shown in FIG. Alternatively, the number of learning images 101 displayed in the display field 94 is calculated based on the "learning image group" field of the record corresponding to the target operation in the history information 125 shown in FIG.

In the display column 95, the allowable missed rate set for determining the determination parameter 56 is displayed.

By looking at the screen 80 shown in FIG. 13 or the screen 90 shown in FIG. 14, the user can grasp changes in the evaluation results of the performance of the learning model due to operations performed in the past.

Note that the screen 80 shown in FIG. 13 or the screen 90 shown in FIG. 14 includes a button 89 for closing the screen. When button 89 is pressed, output unit 17 closes the screen.

(C-6. Setting section)
The setting unit 18 sets the specified learning model 5 in the determination device 2 according to instructions from the user.

For example, in response to a rollback execution button 88 included in the screen 80 shown in FIG. 13 or the screen 90 shown in FIG. Transfer to 2. Specifically, when the rollback execution button 88 included in the screen 80 shown in FIG. The model data 123 is transferred to the determination device 2. Similarly, when the rollback execution button 88 included in the screen 90 shown in FIG. is transferred to the determination device 2.

When the storage unit 120 stores the history information 125 shown in FIG. You can transfer it to

When the storage unit 120 stores the history information 125 shown in FIG. The machine learning used is instructed to the learning unit 13. The setting unit 18 may transfer the model data 123 that defines the learning model 5 generated by the learning unit 13 to the determination device 2 .

Thereby, the user can set the learning model 5 having an excellent performance evaluation result to the determination device 2 while viewing the screen 80 shown in FIG. 13 or the screen 90 shown in FIG. 14.

<D. Example of software configuration of determination device>
Next, an example of the software configuration of the determination device 2 according to this embodiment will be described using FIG. 15. FIG. 15 is a diagram schematically illustrating an example of the software configuration of the determination device.

As shown in FIG. 15, the determination device 2 includes an acquisition section 21, a generation section 22, a determination section 23, and an output section 24 as software modules. Each software module is realized by the CPU 210 interpreting and executing instructions included in the determination program 221.

The acquisition unit 21 acquires an observation image 201 obtained by observing the workpiece W. The workpiece W appears in the observation image 201.

The generation unit 22 refers to the model data 123 transferred from the learning device 1 and sets the learning model 5. As described above, the learning model 5 is configured to, when an image is input, convert the input image into a feature amount, and generate an image by restoring the input image from the feature amount obtained by the conversion. has been done. The generation unit 22 generates a restored observation image 203 by restoring the observation image 201 by inputting the acquired observation image 201 into the learning model 5 .

The determining unit 23 determines whether the workpiece W shown in the observed image 201 is a good product based on the difference between the observed image 201 and the restored observed image 203. In the present embodiment, the determination unit 23 determines the quality of the work W according to the same method as the evaluation unit 14. That is, the determination unit 23 generates a difference image 205 indicating the difference between the observed image 201 and the restored observed image 203 by calculating the difference between the observed image 201 and the restored observed image 203. The determination unit 23 generates the detected image I14 by applying the determination parameters 56 (threshold values 57, 58) transferred from the learning device 1 to the difference image 205 (see FIG. 7). Then, the determining unit 23 determines whether a defect exists in the workpiece W shown in the observed image 201, depending on whether a white pixel area exists in the detected image I14. The determination unit 23 determines that the workpiece W is defective when there is a defect in the workpiece W, and determines that the workpiece W is a non-defective item when there is no defect in the workpiece W.

The output unit 24 outputs information regarding the determination result of the determination unit 23, that is, the result of determining the quality of the workpiece W.

<E. Learning device processing flow>
The flow of processing of the learning device 1 will be described with reference to FIG. 16. FIG. 16 is a flowchart showing an example of the processing flow of the learning device.

First, the CPU 110 of the learning device 1 determines whether an operation related to machine learning has been received (step S1). If no operation has been accepted (NO in step S1), the process returns to step S1.

If the operation is accepted (YES in step S1), the CPU 110 updates data according to the accepted operation (step S2). For example, when the first operation is received, the CPU 110 updates the learning image group 11. If the second operation is accepted, the CPU 110 updates the evaluation image group 12.

Next, the CPU 110 generates the learning model 5 by performing machine learning using the learning image group 11 (step S3). Subsequently, the CPU 110 evaluates the performance of the generated learning model 5 (step S4). The CPU 110 associates and records the content of the received operation and the evaluation result of the performance of the learning model 5 (step S5).

Next, the CPU 110 determines whether an instruction to output the history information 125 recorded by the recording unit 16 has been received (step S6). If the output instruction has not been received (NO in step S6), the process returns to step S1.

When receiving an output instruction (YES in step S6), CPU 110 outputs history information 125 (step S7). Specifically, CPU 110 causes display device 6 to display a screen showing history information 125.

Next, the CPU 110 determines whether an instruction to end the screen display has been received (step S8). If the termination instruction is received (YES in step S8), the process returns to step S1.

If the termination instruction has not been received (NO in step S8), the CPU 110 determines whether or not an instruction to perform rollback has been received (step S9). If the rollback execution instruction has not been received (NO in step S9), the process returns to step S8.

When receiving the instruction to perform rollback (YES in step S9), the CPU 110 sets the learning model 5 immediately after the specified target operation was executed in the determination device 2 (step S10). Specifically, the CPU 110 transfers model data 123 corresponding to the learning model 5 to the determination device 2. After step 10, the process returns to step S1.

<F. Experimental example>
Next, an experimental example of the present disclosure will be described. Using a general-purpose computer, we conducted an experiment to construct a learning model according to the following experimental conditions.

(Experimental conditions)
・Number of learning images included in the learning image group at the startup stage: 10
・Number of additional learning: 7 times ・Number of images for first evaluation: 42
・Number of images for second evaluation: 3445
・Size of each image: 350 x 350
・Pixel value of each image: 256 gradations ・Pixel value range of difference image: 0 to 255
・Binarization threshold (threshold 57) value: 10 to 70
・Value of threshold (threshold 58) for area (number of pixels): 10 to 1000
・Tolerable missed rate: 0%.

FIG. 17 is a diagram showing evaluation results of the performance of a learning model constructed in the process of repeating machine learning processing according to an experimental example. The field "Learning effect evaluation value of additionally learned image" indicates the learning effect evaluation value for the learning image added to the learning image group. The "performance evaluation value" field indicates the "miss rate" and "overlook rate" which are the performance evaluation results of the learning model. The “determination parameter” field indicates the optimal value of the determination parameter (threshold values 57, 58) determined by the evaluation unit 14.

As shown in FIG. 17, the performance of the learning model is improved from the first to the fifth times by adding learning images. However, after the 6th time, even if learning images are added, the performance does not necessarily improve. Therefore, it is preferable to repeat machine learning multiple times and adopt the learning model with the highest performance.

According to the present disclosure, the evaluation result of the performance of the learning model is recorded for each operation, and the recorded history information is output. Thereby, the user can easily understand at which timing of the operation the learning model becomes optimal. As a result, the learning model to be set in the determination device can be easily adjusted.

<G. Action/Effect>
As described above, the learning device 1 of this embodiment includes the learning section 13, the evaluation section 14, the receiving section 15, the recording section 16, and the output section 17. The learning unit 13 generates a learning model 5 used to determine the attributes of the workpiece W by performing machine learning using a learning image group 11 including one or more learning images 101 in which the workpiece W is captured. do. The evaluation unit 14 evaluates the performance of the learning model 5 by inputting one or more evaluation images 102 included in the evaluation image group 12 to the learning model 5. The reception unit 15 receives a first operation for updating the learning image group 11. The recording unit 16 records, for each first operation, the content of the first operation and the evaluation result for the learning model 5 generated using the updated learning image group 11 in association with each other. The output unit 17 outputs the history information 125 recorded by the recording unit 16.

According to the above configuration, by checking the history information 125 recorded by the recording unit 16, the user can determine which timing of learning among the learning models 5 corresponding to each of a plurality of first operations performed in the past. It is easy to understand whether Model 5 has the highest performance. Thereby, the user can adjust the learning model 5 as appropriate while checking the performance of the learning model 5 for each first operation.

The learning model 5 receives an image of the workpiece W as input, and outputs output information for determining the quality of the workpiece W. The evaluation image group 12 includes one or more first evaluation images 103 in which a workpiece W that is non-defective is captured, and one or more second evaluation images 104 in which a workpiece W that is a defective product is captured. The evaluation unit 14 calculates a first evaluation value and a second evaluation value as evaluation results. The first evaluation value is the first evaluation value of the first evaluation image 103 in which the workpiece W is determined to be a defective product based on the output information obtained by inputting it to the learning model 5, among the one or more first evaluation images 103. It is a value according to 1 percentage. The second evaluation value is a second percentage of the second evaluation images in which the workpiece W is determined to be a good product based on output information obtained by inputting it to the learning model 5 among the one or more second evaluation images 104. The value corresponds to

According to the above configuration, the user can understand the first ratio by checking the first evaluation value, and can understand the second ratio by checking the second evaluation value. Thereby, the user can adjust the learning model 5 so that the first ratio and the second ratio fall within a desired range.

Determination of the quality of the workpiece W using the output information is performed depending on whether the output information satisfies the determination conditions. The evaluation unit 14 determines the determination condition so that the second ratio is equal to or less than a predetermined allowable missed rate.

On the production line, it is necessary to avoid shipping defective products as non-defective products as much as possible. According to the above configuration, the performance of the learning model 5 is evaluated based on the result of the pass/fail determination carried out under the determination conditions such that the second ratio is equal to or less than the predetermined allowable missed rate. Thereby, the user can adjust the learning model 5 to have high performance under the determination condition that the second ratio is equal to or less than the allowable missed rate.

The learning device 1 further includes a setting unit 18 that sets the learning model 5 in the determination device 2 that determines the attributes of the workpiece W. The recording unit 16 further records the updated learning image group 11 for each first operation. The setting unit 18 sets the learning model 5, which is generated using the learning image group 11 recorded in response to the first operation specified from the history information 125, in the determination device 2.

Alternatively, the recording unit 16 further records the learning model 5 generated using the updated learning image group 11 for each first operation. The setting unit 18 sets the learning model 5 recorded in correspondence to the first operation specified from the history information 125 in the determination device 2 .

According to the above configuration, the user can easily set the learning model 5 with high performance to the determination device 2 while viewing the history information.

The receiving unit 15 further receives a second operation for updating the evaluation image group 12. The recording unit 16 further records, for each second operation, the content of the second operation and the result of the evaluation using the updated evaluation image group 12 in association with each other.

According to the above configuration, the user can check the change in the evaluation result of the performance of the learning model 5 when the evaluation image group 12 is updated. As a result, the user can optimize the evaluation image group 12 by checking the evaluation results.

The output unit 17 causes the display device 6 to display a list of the contents of the first operation and the evaluation results for each first operation. Thereby, the user can check the change in performance of the learning model 5 according to the operation by looking at the list.

Alternatively, the evaluation unit 14 calculates evaluation values (for example, a first evaluation value and a second evaluation value) indicating the performance of the learning model 5 as the evaluation result. The output unit 17 causes the display device 6 to display a graph showing the transition of the evaluation value for each first operation. This makes it easier for the user to check changes in the performance of the learning model 5 depending on the operation by looking at the graph.

<H. Modified example>
(H-1. Modification 1)
In the above description, the reception unit 15 receives the first operation and the second operation. The reception unit 15 may further receive a third operation for updating model generation parameters used to generate the learning model 5.

When the receiving unit 15 accepts the third operation, the recording unit 16 records the update contents of the model generation parameters, the performance evaluation result of the learning model 5 generated using the updated model generation parameters, and the learning data. A record is created in which the model data 123 that defines the model 5, the number of learning images 101 included in the current learning image group 11, and the determination parameter 56 determined by the evaluation unit 14 are associated.

Alternatively, the recording unit 16 stores the updated content of the model generation parameters, the updated set of model generation parameters, and the performance evaluation result of the learning model 5 generated using the updated model generation parameters, A list of learning images 101 included in the current learning image group 11 , a list of each of the first evaluation images 103 and second evaluation images 104 included in the current evaluation image group 12 , and the evaluation unit 14 A record may be created in which the determined determination parameters 56 are associated with each other.

According to the first modification, the user can check changes in the evaluation results of the performance of the learning model 5 when the model generation parameters are updated. As a result, the user can optimize the model generation parameters by checking the evaluation results.

(H-2. Modification 2)
The learning unit 13 may divide each learning image 101 into a plurality of patch images. The division method and the number of patch images are not particularly limited and may be determined as appropriate depending on the embodiment. For example, the learning unit 13 may equally divide the learning image 101 into specified numbers in both the vertical and horizontal directions.

The learning unit 13 may generate the learning model 5 for each patch image through the machine learning process described above. That is, the learning unit 13 constructs a learning model 5 trained to convert a patch image into a feature amount, and restore an image suitable for the corresponding patch image from the feature amount obtained by the conversion. As a result, a learning model 5 is generated for each patch image.

When the learning model 5 for each patch image is generated, the evaluation unit 14 divides each evaluation image 102 into a plurality of patch images. The evaluation unit 14 uses the corresponding learning model 5 for each patch image to generate a restored image corresponding to the patch image. The evaluation unit 14 determines the quality of the workpiece W shown in each evaluation image 102 based on the degree of difference between each of the plurality of patch images and the corresponding restored image. The evaluation unit 14 evaluates the performance of the learning model 5 based on the determination result.

§3 Supplementary notes As described above, this embodiment includes the following disclosures.

(Configuration 1)
A learning device (1),
By performing machine learning using a learning image group (11) including one or more learning images (101) in which the target object (W) is captured, a learning unit (13) that generates a learning model (5);
an evaluation unit that evaluates the performance of the learning model (5) by inputting one or more evaluation images (102, 103, 104) included in the evaluation image group (12) to the learning model (5); (14) and
a reception unit (15) that receives a first operation for updating the learning image group (11);
For each of the first operations, the contents of the first operation and the results of the evaluation of the learning model (5) generated using the updated learning image group (11) are recorded in association with each other. a recording section (16);
A learning device (1) comprising an output section (17) that outputs history information (125) recorded by the recording section (16).

(Configuration 2)
The learning model (5) receives an image of the object (W) as input, outputs output information for determining the quality of the object (W),
The evaluation image group (12) includes one or more first evaluation images (103) in which the object (W) is a non-defective item, and one or more first evaluation images (103) in which the object (W) is a defective item. 2 evaluation images (104),
As a result of the evaluation, the evaluation unit (14)
A first evaluation in which the object (W) is determined to be a defective product based on the output information obtained by inputting it into the learning model (5) among the one or more first evaluation images (103). a value corresponding to the first ratio of the image for use (103),
Among the one or more images for second evaluation (104), the image for second evaluation in which the object (W) is determined to be a good product based on the output information obtained by inputting it to the learning model (5). The learning device (1) according to configuration 1, which calculates a value according to a second ratio of the image (104).

(Configuration 3)
The determination of the quality of the object (W) using the output information is carried out depending on whether the output information satisfies the determination condition,
The learning device (1) according to configuration 2, wherein the evaluation unit (14) determines the determination condition such that the second ratio is equal to or less than a predetermined threshold.

(Configuration 4)
further comprising a setting unit (18) that sets the learning model (5) to the determination device (2) that determines the attribute of the object (W),
The recording unit (16) further records the updated learning image group (11) for each of the first operations,
The setting unit (18) sets the learning model (5) generated using the learning image group (11) recorded in response to a first operation specified from the history information (125). The learning device (1) according to any one of configurations 1 to 3, wherein the determination device (2) is configured to set the following.

(Configuration 5)
further comprising a setting unit (18) that sets the learning model (5) to the determination device (2) that determines the attribute of the object (W),
The recording unit (16) further records the learning model (5) generated using the updated learning image group (11) for each of the first operations,
From configuration 1, the setting unit (18) sets the learning model (5) recorded in response to a first operation specified from the history information (125) in the determination device (2). 3. The learning device (1) according to any one of 3.

(Configuration 6)
The reception unit (15) further receives a second operation for updating the evaluation image group (12),
The recording unit (16) further records, for each second operation, the content of the second operation and the result of the evaluation using the updated evaluation image group (12) in association with each other. , the learning device (1) according to any one of configurations 1 to 5.

(Configuration 7)
The learning unit (13) generates the learning model (5) using model generation parameters,
The reception unit (15) further receives a third operation for updating the model generation parameters,
The recording unit (16) further records, for each third operation, the content of the third operation and the result of the evaluation of the learning model (5) generated using the updated model generation parameters. The learning device (1) according to any one of configurations 1 to 6, wherein the learning device (1) records information in association with each other.

(Configuration 8)
The learning device (1) according to configuration 1, wherein the output unit (17) causes a display device (6) to display a list of the content of the first operation and the evaluation result for each first operation.

(Configuration 9)
The evaluation unit (14) calculates an evaluation value indicating the performance of the learning model (5) as a result of the evaluation,
The learning device according to configuration 1, wherein the output unit (17) causes a display device (6) to display a graph showing the transition of the evaluation value for each of the first operations.

(Configuration 10)
By performing machine learning using a learning image group (11) including one or more learning images (101) in which the target object (W) is captured, a step of generating a learning model (5);
evaluating the performance of the learning model (5) by inputting one or more evaluation images (102) included in the evaluation image group (12) to the learning model (5);
accepting an operation to update the learning image group (11);
For each of the operations, the content of the operation and the result of the evaluation of the learning model (5) generated using the updated learning image group (11) are recorded in association with each other;
A learning method comprising the step of outputting recorded history information (125).

(Configuration 11)
A program that causes a computer to execute the learning method according to configuration 10.

Although the embodiments of the present invention have been described, the embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, and it is intended that all changes within the meaning and range equivalent to the claims are included.

1 Learning device, 2 Judgment device, 3 Camera, 5 Learning model, 6, 9 Display device, 7, 8 Input device, 11 Learning image group, 12 Evaluation image group, 13 Learning section, 14 Evaluation section, 15 Reception section , 16 recording unit, 17 output unit, 18 setting unit, 21 acquisition unit, 22 generation unit, 23 determination unit, 24 output unit, 51 input layer, 52 intermediate layer, 53 output layer, 55 determination parameter candidate, 56 determination parameter, 57, 58 Threshold, 60, 70, 80, 90 Screen, 61, 71 Candidate list, 62, 83 Learning image list, 63, 74, 86 Area, 64, 75a, 75b Add button, 65, 77a, 77b Delete button, 66 Learning execution button, 72, 85 Good product image list, 73, 84 Defective product image list, 76 Movement button, 78 Evaluation execution button, 81 Operation history table, 82, 92 Cursor, 87, 93-95 Display field, 88 Rollback Execution button, 89 Button, 91 Operation history graph, 101 Learning image, 102 Evaluation image, 103 First evaluation image, 104 Second evaluation image, 105 First restored image, 106 Second restored image, 107, 108 ,205,I12 Differential image, 110,210 CPU, 111,211 RAM, 112,212 ROM, 113,213 Communication interface, 114,215 Input interface, 115,216 Display controller, 116,217 Drive, 120,220 Storage unit , 122 learning program, 123 model data, 124 evaluation program, 125 history information, 130, 230 storage medium, 201 observed image, 203 restored observed image, 214 external interface, 221 determination program, I10 image, I11 restored image, I13 binary image, I14 detection image, L defect, SYS system, W work.

Claims (11)

A learning device,
a learning unit that generates a learning model used to determine attributes of the object by performing machine learning using a learning image group including one or more learning images of the object;
an evaluation unit that evaluates the performance of the learning model by inputting one or more evaluation images included in the evaluation image group to the learning model;
a reception unit that accepts a first operation to update the learning image group;
a recording unit that records, for each of the first operations, the content of the first operation and the result of the evaluation of the learning model generated using the updated learning image group;
an output unit that outputs history information recorded by the recording unit ,
The first operation includes an additional operation of adding a learning image to the learning image group,
The content of the first operation includes identification information for identifying the added learning image for the additional operation . The learning model receives an image of the object as input, outputs output information for determining the quality of the object,
The evaluation image group includes one or more first evaluation images in which the object is a non-defective product and one or more second evaluation images in which the object is a defective product,
As a result of the evaluation, the evaluation unit:
Among the one or more first evaluation images, the object is determined to be a defective product based on the output information obtained by inputting the learning model to a first percentage of the first evaluation images. value and
A value corresponding to a second ratio of second evaluation images in which the object is determined to be a good product based on the output information obtained by inputting the object to the learning model, among the one or more second evaluation images. The learning device according to claim 1, which calculates. The determination of the quality of the object using the output information is performed depending on whether the output information satisfies a determination condition,
The learning device according to claim 2, wherein the evaluation unit determines the determination condition such that the second ratio is equal to or less than a predetermined threshold. further comprising a setting unit that sets the learning model to a determination device that determines attributes of the object;
The recording unit further records the updated learning image group for each of the first operations,
From claim 1, wherein the setting unit sets, in the determination device, the learning model generated using the learning image group recorded in response to a first operation specified from the history information. 3. The learning device according to any one of 3. further comprising a setting unit that sets the learning model to a determination device that determines attributes of the object;
The recording unit further records, for each of the first operations, the learning model generated using the updated learning image group,
The learning device according to any one of claims 1 to 3, wherein the setting unit sets the learning model recorded in response to a first operation specified from the history information to the determination device. . The reception unit further receives a second operation for updating the evaluation image group,
5. The recording unit further records, for each second operation, the content of the second operation and the result of the evaluation using the updated evaluation image group in association with each other. The learning device according to any one of the above. The learning unit generates the learning model using model generation parameters,
The reception unit further receives a third operation for updating the model generation parameters,
The recording unit further records, for each third operation, the content of the third operation in association with the evaluation result for the learning model generated using the updated model generation parameters. The learning device according to any one of claims 1 to 6. The learning device according to claim 1, wherein the output unit causes a display device to display a list of the contents of the first operation and the results of the evaluation for each of the first operations. The evaluation unit calculates an evaluation value indicating the performance of the learning model as a result of the evaluation,
The learning device according to claim 1, wherein the output unit causes a display device to display a graph showing the transition of the evaluation value for each of the first operations. generating a learning model used to determine the attributes of the object by performing machine learning using a learning image group including one or more learning images of the object;
evaluating the performance of the learning model by inputting one or more evaluation images included in the evaluation image group to the learning model;
accepting an operation to update the learning image group;
For each operation, the content of the operation and the result of the evaluation of the learning model generated using the updated training image group are recorded in association with each other;
and a step of outputting recorded history information ,
The operation includes an additional operation of adding a learning image to the learning image group,
The content of the operation includes identification information for identifying the added learning image for the additional operation . A program that causes a computer to execute the learning method according to claim 10.

Images