JP7635661B2 - Determination device, determination method, and determination program - Google Patents

Description

The present invention relates to a determination device, a determination method, and a determination program.

Patent Document 1 describes a cell evaluation device including: an image reader that reads in a plurality of images of a plurality of cells cultured in a culture vessel in a time series; a feature calculation unit that obtains, from the images, a plurality of different feature amounts that indicate a plurality of different morphological features of the cells for each of the cells included in the images; a feature determination unit that determines, for each of the feature amounts corresponding to each of the images, whether it is appropriate to evaluate a characteristic to be evaluated for the cells using the feature amounts; a continuity determination unit that determines, for each of the feature amounts, whether the determination results by the feature determination unit for the feature amounts are continuous in the time series; and a computational model construction unit that combines the feature amounts obtained by the feature calculation unit and the determination results by the continuity determination unit to construct a computational model for evaluating the characteristics of the cells (claim 1). Patent Document 2 describes an image processing device that includes a CNN processing unit that uses a convolutional neural network to extract image features of the bright field image and the fluorescent image and output identification information related to the cells (claim 1). Furthermore, Patent Document 2 states that "the identification information relating to the cell includes information relating to the type, morphology, or distribution of the cell, or information relating to the type, morphology, or distribution of biological material contained within the cell" (Claim 2).
[Prior Art Literature]
[Patent Documents]
[Patent Document 1] JP 2011-229410 A [Patent Document 2] JP 2018-180635 A

In a first aspect of the present invention, a determination device is provided. The determination device may include an acquisition unit that acquires a captured image of a field of view including cells. The determination device may include a calculation unit that calculates the degree of abnormality of the captured image based on one or more reference images of normally cultured cells. The determination device may include a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality.

The judgment device may be equipped with an analysis processing unit that generates an analysis image that displays the distribution of influences on the judgment on the captured image.

The calculation unit may include a reconstructed image generation unit that generates a reconstructed image by reconstructing the captured image so as to resemble one or more reference images. The calculation unit may include an anomaly calculation unit that calculates the anomaly based on the difference between the captured image and the reconstructed image.

The reconstructed image generating unit may extract features of each of one or more reference images, input the captured image to a reconstruction model trained to restore each reference image from the extracted features, and generate a reconstructed image.

The judgment device may include a learning processing unit that performs learning processing of the reconstruction model using one or more reference images captured of a field of view containing cells that have been judged to have been cultured normally.

The learning processing unit may add a captured image that is not determined by the determination unit to have an abnormality in the cell culture as a new reference image to a set of one or more reference images, and perform learning processing of the reconstruction model.

The determination device may include a plurality of calculation units. A plurality of reconstructed image generation units included in the plurality of calculation units may input the captured image to a plurality of different reconstructed models, respectively, to generate a plurality of reconstructed images. A plurality of abnormality calculation units included in the plurality of calculation units may calculate the abnormality degree for each of the plurality of reconstructed images. The determination unit may output a determination result that integrates the abnormality degrees of each of the plurality of reconstructed images.

The calculation unit may have a feature calculation unit that calculates a feature of the captured image. The calculation unit may have an anomaly calculation unit that calculates an anomaly based on the feature of the captured image and each of the reference features of one or more reference images.

The acquisition unit may acquire images of the cell culture process captured in time series. The calculation unit may calculate the degree of abnormality based on one or more reference images of the normal cell culture process captured in time series.

The determination device may include a preprocessing unit that performs preprocessing on the captured image. The calculation unit may calculate the degree of abnormality using the captured image that has been preprocessed.

The pre-processing may include at least one of a 3D reconstruction process or an edge enhancement process of the cell or cell cluster image.

The acquisition unit may acquire a captured image for each of the multiple fields of view. The determination device may include multiple calculation units. The calculation units may calculate the degree of abnormality of each of the captured images for the multiple fields of view. The determination unit may output a determination result that integrates the degree of abnormality of each of the captured images for the multiple fields of view.

The determination unit may output a determination result that integrates the degree of abnormality for each culture plate or well.

The determination unit may output a determination result that integrates the degree of abnormality for multiple culture plates or wells.

The acquisition unit may further acquire the cell culture conditions. The calculation unit may calculate the degree of abnormality based on one or more reference images of cells normally cultured under culture conditions corresponding to the acquired culture conditions.

According to a second aspect of the present invention, there is provided a determination method. The determination method may include a determination device acquiring a captured image of a field of view including cells. The determination method may include a determination device calculating a degree of abnormality of the captured image based on one or more reference images of normally cultured cells. The determination method may include a determination device determining, based on the degree of abnormality, whether or not there is an abnormality in the culture of the cells captured in the captured image.

According to a third aspect of the present invention, there is provided a judgment program executed by a computer. The judgment program may cause the computer to function as an acquisition unit that acquires a captured image of a field of view including cells. The judgment program may cause the computer to function as a calculation unit that calculates the degree of abnormality of the captured image based on one or more reference images of normally cultured cells. The judgment program may cause the computer to function as a judgment unit that judges whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality.

Note that the above summary of the invention does not list all of the features of the present invention. Also, subcombinations of these features may also be inventions.

1 shows a configuration of a determination device 10 according to the present embodiment. 4 shows a determination flow of the determination device 10 according to the present embodiment. 4 shows a learning flow of the determination device 10 according to the present embodiment. 13 shows the configuration of a calculation unit 415 according to a first modified example of the present embodiment. 13 shows an operation flow of a calculation unit 415 according to a first modified example of the present embodiment. 6 shows the configuration of a determination device 600 according to a second modified example of the present embodiment. 7 shows the configuration of a determination device 700 according to a third modified example of the present embodiment. 13 shows the configuration of a determination device 800 according to a fourth modified example of the present embodiment. 13 shows a determination flow of a determination device 800 according to a fourth modified example of the present embodiment. 13 shows the configuration of a determination device 1000 according to a fifth modified example of the present embodiment. 13 shows a determination flow of a determination device 1000 according to a fifth modified example of the present embodiment. 22 illustrates an example computer 2200 in which aspects of the present invention may be embodied, in whole or in part.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

Figure 1 shows the configuration of a determination device 10 according to this embodiment. Many aspects of bioprocesses such as cell culture have not been fully elucidated. For this reason, even if the same protocol is used, variations in the culture process can occur due to, for example, differences in the technique of the experimenter, differences in the equipment when changing the equipment, the degree of deterioration of the raw cells, contamination, and other unexplained factors.

When using a model trained to extract features from images of normally cultured cells, if there is an abnormality in the culture of the target cells, there is a possibility that incorrect features will be extracted from the image resulting from the abnormality in the culture. The determination device 10 determines whether the cell was cultured normally or whether there was an abnormality, making it possible to indicate whether or not the cell can be used as a normally cultured cell, i.e., whether or not an image of the cell can be used to extract the cell's features.

The determination device 10 determines whether or not there is an abnormality in the culture by using an image of a measurement target 20 including cells. The determination device 10 may be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system to which multiple computers are connected. Such a computer system is also a computer in a broad sense. The determination device 10 may be implemented by a virtual computer environment in which one or more can be executed within a computer. Alternatively, the determination device 10 may be a dedicated computer designed for determining the culture state of cells, or may be dedicated hardware realized by a dedicated circuit. When a computer is used, the determination device 10 is realized by executing a determination program by the computer. The determination device 10 includes an imaging unit 100, a captured image storage unit 105, an acquisition unit 110, a calculation unit 115, a determination unit 130, an analysis processing unit 132, a result output unit 135, and a model generation unit 137.

The imaging unit 100 captures an image of a field of view including cells. The captured image storage unit 105 is connected to the imaging unit 100 and stores the captured image captured by the imaging unit 100. The acquisition unit 110 is connected to the captured image storage unit 105 and acquires the captured image of the field of view including cells from the captured image storage unit 105. Note that in other embodiments, the determination device 10 may not include the imaging unit 100 and the captured image storage unit 105, and may acquire the captured image from an external imaging device or storage device, etc.

The calculation unit 115 is connected to the acquisition unit 110. The calculation unit 115 calculates the degree of abnormality of the captured image based on one or more reference images of normally cultured cells. Here, the "degree of abnormality" is an index value indicating the degree to which the captured image deviates from a standard (standard) defined using one or more reference images. In this embodiment, the calculation unit 115 has a reconstructed image generation unit 120 and an abnormality calculation unit 125.

The reconstructed image generating unit 120 generates a reconstructed image by reconstructing the captured image acquired by the acquiring unit 110 so as to resemble one or more reference images. This allows the reconstructed image generating unit 120 to output a reconstructed image from the captured image in which image features caused by culture abnormalities, such as those not seen in one or more reference images of normally cultured cells, have been removed or attenuated.

The degree of abnormality calculation unit 125 is connected to the reconstructed image generation unit 120. The degree of abnormality calculation unit 125 calculates the degree of abnormality of the captured image based on the difference between the captured image and the reconstructed image. Here, since the reconstructed image is an image from which image features caused by culture abnormalities have been removed, the greater the image features caused by culture abnormalities are contained in the captured image, the greater the difference between the captured image and the reconstructed image. Therefore, in such a case, the "degree of abnormality" is a value indicating a greater deviation from the standard.

The determination unit 130 is connected to the calculation unit 115. The determination unit 130 determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality calculated by the calculation unit 115.

The analysis processing unit 132 is connected to the determination unit 130. The analysis processing unit 132 generates an analysis image on the captured image that displays the distribution of influence on the determination by the determination unit 130. The analysis image indicates which area in the captured image contributed to the determination that there was an abnormality in the cell culture. For example, the analysis processing unit 132 generates an analysis image on the captured image that displays the difference from the reconstructed image. Note that in other embodiments, the determination device 10 does not need to include the analysis processing unit 132 and does not need to have the function of generating an analysis image.

The result output unit 135 is connected to the analysis processing unit 132. The result output unit 135 outputs the judgment result by the judgment unit 130 and the analysis image generated by the analysis processing unit 132. For example, the result output unit 135 performs a display process to display the judgment result and the analysis image on a display device or terminal used by the user. The result output unit 135 may store the judgment result and the analysis image in a storage device.

The model generation unit 137 is connected to the acquisition unit 110. The model generation unit 137 generates a reconstruction model using one or more reference images of a field of view including cells determined to have been normally cultured. The model generation unit 137 according to this embodiment selects a reference image from the captured images acquired by the acquisition unit 110 and uses the reference image to generate a reconstruction model. Note that in other embodiments, the determination device 10 does not include the model generation unit 137, and the result output unit 135 may generate a reconstruction image using a predetermined reconstruction model or an externally provided reconstruction model.

The model generation unit 137 has a reference image storage unit 145, a learning processing unit 150, and a model storage unit 155. The reference image storage unit 145 stores one or more reference images of normally cultured cells.

The learning processing unit 150 is connected to the acquisition unit 110, the judgment unit 130, and the reference image storage unit 145. The learning processing unit 150 performs learning processing of a reconstruction model used by the reconstruction image generation unit 120 to reconstruct a captured image, using one or more reference images stored in the reference image storage unit 145. In this embodiment, the learning processing unit 150 adds, as new reference images, captured images acquired by the acquisition unit 110 that have not been judged by the judgment unit 130 to have an abnormality in cell culture, to a set of one or more reference images stored in the reference image storage unit 145. Then, the learning processing unit 150 performs learning processing of the reconstruction model using one or more reference images to which the new reference image has been added. Note that in other embodiments, the learning processing unit 150 may not have the function of online learning to update the reconstruction model by adding a new reference image to the reference image storage unit 145.

The model storage unit 155 is connected to the learning processing unit 150. The model storage unit 155 stores the reconstructed model generated by the learning processing unit 150. The model storage unit 155 then supplies the reconstructed model to the reconstructed image generation unit 120 in the calculation unit 115.

Figure 2 shows the judgment flow of the judgment device 10 according to this embodiment. In step 200 (S200), the imaging unit 100 captures an image of a field of view including cells. The imaging unit 100 may be, for example, a microscope with an imaging function. In this embodiment, the imaging unit 100 captures a bright field image of a field of view including cells in the measurement target 20. The imaging unit 100 may capture, for example, a phase contrast image of a field of view including cells, a stained image of a field of view including stained cells, or an observation image using various other techniques. The imaging unit 100 stores the captured image in the captured image storage unit 105. In S210, the acquisition unit 110 acquires the captured image from the captured image storage unit 105.

In S220, the reconstructed image generating unit 120 generates a reconstructed image by reconstructing the captured image to resemble one or more reference images. In this embodiment, the reconstructed image generating unit 120 generates a reconstructed image by reconstructing the captured image using a reconstruction model generated by the model generating unit 137 and stored in the model storage unit 155. Here, the learning processing unit 150 in the model generating unit 137 extracts features of each reference image among one or more reference images, and generates a reconstruction model by learning so as to restore each reference image from the extracted features. The reconstructed image generating unit 120 inputs the captured image into such a reconstruction model to generate a reconstructed image. Note that the reconstructed image generating unit 120 may reconstruct the captured image before reconstruction to generate a reconstructed image with the same resolution, or may generate a reconstructed image with a different resolution. In this way, the reconstructed image generating unit 120 reconstructs the captured image to resemble one or more reference images, thereby generating an image that serves as a reference (standard) in accordance with one or more reference images in the situation in which the captured image was captured.

In S230, the abnormality degree calculation unit 125 calculates the abnormality degree of the captured image based on the difference between the captured image and the reconstructed image. In this embodiment, the abnormality degree calculation unit 125 uses the sum of squared errors obtained by summing the squared errors of each pixel of the captured image and the reconstructed image as an index representing the abnormality degree. That is, the abnormality degree a(t) of a captured image _x (t) captured at a certain time t is expressed by the following formula (1), where the subscript i is a variable representing each pixel in the image.

The degree of abnormality a(t) expressed by formula (1) is an index value that increases as the difference between the captured image and the reconstructed image increases. The degree of abnormality a(t) calculated in this manner is an index value that indicates the degree of deviation of the captured image based on a reconstructed image that conforms to one or more reference images in the situation in which the captured image was captured. Alternatively, the degree of abnormality may be an index value that decreases as the difference between the captured image and the reconstructed image increases. Note that when the resolutions of the captured image and the reconstructed image are different, the degree of abnormality calculation unit 125 may calculate the degree of abnormality a(t) by formula (1) using the difference between pixels at corresponding positions.

Furthermore, the abnormality degree calculation unit 125 may calculate the abnormality degree using a calculation method different from the above. The abnormality degree calculation unit 125 may use any statistical quantity that changes depending on the difference between the captured image and the reconstructed image, such as the maximum error among the errors of each pixel in the captured image _x (t) and the reconstructed image x rec (t).

In S240, the determination unit 130 determines whether or not there is an abnormality in the culture of the cells captured in the captured image, based on the degree of abnormality calculated by the calculation unit 115. The determination unit 130 determines that there is an abnormality in the culture of the cells when the degree of abnormality falls outside the normal range. When using a degree of abnormality that increases as the difference between the captured image and the reconstructed image increases, the determination unit 130 may determine that there is an abnormality in the culture of the cells when the degree of abnormality a(t) exceeds a predetermined threshold value.

When it is determined in S240 that the cell culture was normal, the learning processing unit 150 in the model generation unit 137 adds the captured image for which it is determined that the cell culture was normal as a new reference image to the set of one or more reference images stored in the reference image storage unit 145 in S270. The determination device 10 then proceeds to S260.

When it is determined in S240 that there is an abnormality in the cell culture, the analysis processing unit 132 generates an analysis image in S250 that indicates which area in the captured image contributed to the determination that there was an abnormality in the cell culture. The analysis processing unit 132 may generate an analysis image that indicates the part in the captured image that influenced the determination and the degree of influence. In this embodiment, the analysis processing unit 132 generates an analysis image that displays the difference from the reconstructed image on the captured image. The analysis processing unit 132 may generate an analysis image using a heat map or contour lines, etc. For example, the analysis processing unit 132 reflects the difference from the reconstructed image on the captured image by adding a color corresponding to the magnitude of the difference in pixel value between the captured image and the reconstructed image to the pixel value of the captured image for each pixel of the captured image. Also, for example, the analysis processing unit 132 may add a contour line indicating the difference in pixel value between the captured image and the reconstructed image to the captured image.

In S260, the result output unit 135 outputs the result of the determination made by the determination unit 130, i.e., whether or not there was an abnormality in the culture of the cells captured in the captured image. Furthermore, if it is determined that there was an abnormality in the culture of the cells, the result output unit 135 outputs the analysis image generated by the analysis processing unit 132.

According to the determination device 10 described above, by comparing a captured image with a reconstructed image obtained by reconstructing the captured image using a reconstruction model trained using one or more reference images, the degree of abnormality of the captured image based on one or more reference images can be calculated, and it can be determined whether or not there is an abnormality in the cell culture. Furthermore, according to the determination device 10, since it is possible to display on the captured image an area where the difference between the captured image and the reconstructed image is relatively large, it is possible to identify the areas that should be focused on in the analysis of the captured image that is determined to have an abnormality in the cell culture.

Figure 3 shows the learning flow of the determination device 10 according to this embodiment. Prior to the learning process, the reference image storage unit 145 stores one or more reference images of normally cultured cells. Here, the one or more reference images may be images of normally cultured cells from the start to the end of the culture process, or may be images of normally cultured cells at any point in time from the start to the end of the culture process.

The reference image storage unit 145 may store one or more reference images selected by the user as images of normally cultured cells prior to learning as an initial set of reference images. In addition, the learning processing unit 150 may select, from among the images acquired by the acquisition unit 110, images for which the determination unit 130 has determined that the cell culture was normal, and store these in the reference image storage unit 145.

Between S300, which corresponds to the start of the loop, and S320, which corresponds to the end of the loop, the learning processing unit 150 performs a loop process that repeats the process between S300 and S320 for each reference image stored in the reference image storage unit 145. In S310, the learning processing unit 150 extracts features of the reference image to be learned, and generates or updates a reconstruction model stored in the model storage unit 155 so as to restore the reference image from the extracted features.

As an example, the reconstruction model may be an autoencoder that includes an encoder that compresses an input reference image into a feature vector with a smaller number of dimensions, and a decoder that restores the reference image from the feature vector, and that trains the encoder and decoder so that the restored reference image more closely resembles the input reference image. The encoder and decoder may include neural networks such as a convolutional neural network (CNN) and a fully connected neural network. The learning processing unit 150 may perform learning processing of the encoder and decoder by an error backpropagation method using the error between the input reference image and the restored reference image.

The learning processing unit 150 may also use a Generative Adversarial Network (GAN) to compress the input image into a feature vector with a smaller number of dimensions and generate a reconstruction model that reconstructs the image using the feature vector. For example, in a system including a generator that generates an image reconstructed from a reference image using the reconstruction model and a discriminator that discriminates whether the image is reconstructed, the learning processing unit 150 performs a learning process for the discriminator to improve the discrimination accuracy of the reference image and the image reconstructed from the reference image, and also performs a learning process for the generator to reduce the discrimination accuracy of the discriminator. The learning processing unit 150 stores the reconstruction model used in the generator learned in this way in the model storage unit 155.

In S330, the learning processing unit 150 determines whether the learning process is complete. For example, the learning processing unit 150 determines that the learning process is complete on the condition that the error between an image reconstructed from a reference image using a reconstruction model and the original reference image is less than a predetermined threshold. The learning processing unit 150 may also determine that the learning process is complete in response to the learning process from S300 to S320 being repeated a predetermined number of times. If it is determined that the learning process is not complete, the learning processing unit 150 advances the process to S300 to continue the learning process.

According to the learning processing unit 150 described above, it is possible to perform a learning process of a reconstruction model that reconstructs a captured image to resemble one or more reference images. Specifically, the learning processing unit 150 trains the reconstruction model to reproduce the features of the reference image as much as possible, but does not train the reconstruction model to reproduce features that are not included in the reference image. Therefore, the reconstruction model reproduces in the reconstructed image features that are included in the captured image and are similar to the reference image, and is hardly able to reproduce in the reconstructed image features that are included in the captured image and are dissimilar to the reference image. Therefore, the reconstruction model can remove or attenuate features in the captured image that are not seen in the reference image obtained by photographing normally cultured cells, and convert the captured image into a reconstructed image that is more similar to the reference image.

In addition, the learning processing unit 150 may perform online learning of the reconstruction model by performing the process of S310 on the new reference image each time it receives a captured image that is not determined to have had an abnormality in the cell culture as a new reference image.

In addition, the determination device 10 may determine the culture state of the cells using a time series of images taken at multiple timings, instead of an image taken at a single time t. In this case, the imaging unit 100 stores the images taken in time series of the cell culture process in the image storage unit 105 in S200 of FIG. 2, and the acquisition unit 110 acquires the time series of images in S210 of FIG. 2.

The calculation unit 115 may calculate the degree of abnormality of the captured image based on one or more reference images captured in time series during a normal cell culture process. Here, the acquisition unit 110 may use, as the captured images in time series and the reference image, a multi-channel image in which pixel values at each of multiple timings are encoded into different channels for each pixel.

Figure 4 shows the configuration of a calculation unit 415 according to a first modified example of this embodiment. In this modified example, a calculation unit 415 is used instead of the calculation unit 115 in the determination device 10 shown in Figure 1. In this modified example, the components other than the calculation unit 115 in the determination device 10 in Figure 1 are basically the same, so the following description will be omitted except for the differences.

The calculation unit 415 is connected to the acquisition unit 110 and the model generation unit 137. The calculation unit 415 calculates the degree of abnormality of the captured image based on one or more reference images of normally cultured cells. The calculation unit 415 has a feature calculation unit 420 and an abnormality calculation unit 425. The feature calculation unit 420 calculates the feature of the captured image acquired by the acquisition unit 110.

The abnormality degree calculation unit 425 is connected to the feature calculation unit 420. The abnormality degree calculation unit 425 calculates the abnormality degree based on the feature of the captured image and the respective reference feature of one or more reference images. This allows the determination device 10 to determine whether or not there is an abnormality in the culture of the cells captured in the captured image based on the difference in the feature from one or more reference images capturing normally cultured cells.

Figure 5 shows the operation of the calculation unit 415 according to a first modified example of this embodiment. In this modified example, the determination device 10 executes S520 and S530 instead of S220 and S230 in Figure 2.

In S520, the feature calculation unit 420 calculates the feature of the captured image. As an example, the feature calculation unit 420 uses a multilayer perceptron, a CNN, a fully connected neural network, or another neural network to calculate a feature obtained by nonlinearly mapping the captured image into a lower dimensional space. The feature calculation unit 420 may use a machine learning model other than a neural network to calculate a feature of a lower dimension than the captured image. In addition, the feature calculation unit 420 may calculate the feature of the captured image using a conventional feature extraction method in image analysis, such as SURF or AKAZE. The feature shown above may be data in a format different from that of an image that includes a luminance value for each pixel.

In S530, the abnormality degree calculation unit 425 calculates the abnormality degree based on the feature amount of the captured image and each of the reference feature amounts of one or more reference images. For example, the abnormality degree calculation unit 425 calculates the degree of difference between the feature amount h(t) of the captured image and an average reference feature amount obtained by averaging the reference feature amounts of one or more reference images as the abnormality degree. In this case, the abnormality degree calculation unit 425 may calculate the abnormality degree a(t) using the following formula (2). Here, μ is the average reference feature amount, and Σ is the covariance of the reference feature amount.

Such an abnormality degree a(t) is calculated by using the degree of difference between the reference features of the captured image and each of the reference images to determine the degree to which the situation captured in the captured image deviates from one or more reference images. In this way, the abnormality degree calculation unit 425 may calculate the abnormality degree of the captured image using one or more reference images as a reference.

The anomaly calculation unit 425 may calculate the anomaly using a calculation method different from the above. For example, the anomaly calculation unit 425 generates a probability distribution model p(t) of the reference feature from one or more sets of reference feature values {h _i (t)} (the subscript i indicates the number of the reference feature value) in one or more reference images, and calculates the anomaly a(t) based on the likelihood of the feature value h(t) of the captured image in the probability distribution model p(t). The anomaly calculation unit 425 may use a negative logarithmic likelihood as the anomaly. The anomaly a(t) calculated in this way is smaller when it is closer to normal (when the likelihood is large) and is larger when it is closer to abnormal (when the likelihood is small).

The calculation unit 415 described above can use the features of the captured image and one or more reference images to calculate the degree of abnormality of the captured image based on one or more reference images, and determine whether or not there is an abnormality in the cell culture. When the calculation unit 415 is used, the model generation unit 137 may generate a feature extraction model used by the feature calculation unit 420 using the method exemplified below.

The learning processing unit 150 in the model generation unit 137 according to this modified example may extract features from various captured images of cultured cells, and generate a machine learning model by learning that restores the captured images from the extracted features. The captured images used for this learning may include images of cells with abnormalities in culture. The learning processing unit 150 may then extract a portion of the captured images in this machine learning model that extracts features, and store this in the model storage unit 155 as a feature extraction model. For example, the learning processing unit 150 may generate an autoencoder by learning using various captured images, and extract the encoder portion of the autoencoder as the feature extraction model.

Alternatively, the determination device 10 may use a trained model in another domain as the feature extraction model, such as a network up to the intermediate feature of a deep learning model created for general image recognition (classification, object detection). The learning processing unit 150 may use learning data in which each captured image of cultured cells is manually labeled as to whether or not there was an abnormality in the culture, to generate a neural network that predicts whether or not there was an abnormality in the culture from the captured image, and extract the intermediate stage of this neural network as the feature extraction model. The learning processing unit 150 may also use similar learning data to perform a learning process for the feature extraction model so that the feature amounts output by the feature extraction model in response to each captured image associated with normal culture are closer to the feature amounts output by the feature extraction model in response to each captured image associated with abnormal culture, and the feature amounts output by the feature extraction model in response to the captured image associated with normal culture are further apart from the feature amounts output by the feature extraction model in response to the captured image associated with abnormal culture.

In addition, when the calculation unit 415 is used, the analysis processing unit 132 generates an analysis image on the captured image that displays the influence of each region on the difference between the feature amount and the reference feature amount. For example, the analysis processing unit 132 may calculate the gradient of the abnormality degree calculation for the captured image input to the feature amount calculation unit 420, and generate an analysis image in which the region in the captured image with a larger gradient is emphasized. In addition, the analysis processing unit 132 may generate an analysis image for each of a plurality of partial regions in the captured image based on the difference between the abnormality degree calculated in response to changing the partial region in the captured image and the abnormality degree calculated from the captured image. For example, if the abnormality degree calculated from an image in which a certain partial region in the captured image is masked, for example, becomes a value that is much smaller (a value with a small deviation from the standard) than the abnormality degree calculated from the captured image itself, the analysis processing unit 132 can determine that the influence of the partial region is large. Alternatively, the analysis processing unit 132 may generate an analysis image that emphasizes areas on the captured image that contribute more to the error between the feature amount h(t) of the captured image and the average reference feature amount μ of one or more reference images. In this case, the analysis processing unit 132 can use a method such as GradCAM to identify the degree of influence of each area on the difference between the feature amount and the reference feature amount.

When using time-series captured images, the feature amount calculation section 420 in the calculation section 415 may extract feature amounts from the time-series captured images, and the abnormality degree calculation section 425 in the calculation section 415 may calculate the abnormality degree based on the feature amounts of the time-series captured images and one or more reference feature amounts extracted from one or more time-series reference images. In this case, the feature amount calculation section 420 may extract the feature amounts and reference feature amounts of the time-series captured images and the time-series reference images using, for example, a recurrent neural network (RNN) as a feature extraction model.

Figure 6 shows the configuration of a determination device 600 according to a second modified example of this embodiment. The determination device 600 according to this modified example has a configuration in which a preprocessing unit 610 is added to the determination device 10 shown in Figure 1. Note that components in Figure 6 that are given the same reference numerals as those in Figure 1 are the same as those in Figure 1, and therefore will not be described below except for the differences.

The determination device 600 further includes a preprocessing unit 610 connected to the acquisition unit 110. The preprocessing unit 610 performs preprocessing on the captured image and supplies it to the calculation unit 115 and the model generation unit 137. When the imaging unit 100 can capture multiple captured images while changing the focal position in the depth direction of the field of view, the preprocessing unit 610 may use these multiple captured images to perform 3D reconstruction processing of an image of a cell or cell cluster (such as a colony, spheroid, or organoid). In other words, the preprocessing unit 610 may reconstruct a 3D captured image in which the cell or cell cluster is converted into 3D data from the multiple captured images sliced in the depth direction. This allows the calculation unit 115 to calculate the degree of abnormality using the 3D captured images.

The pre-processing unit 610 may also perform edge enhancement processing on the image of the cell or cell aggregate. For example, the pre-processing unit 610 may output a captured image acquired by the acquisition unit 110, in which three-dimensional information is convolved with two-dimensional information using a phase contrast method or the like to enhance edges. The model generation unit 137 of the determination device 600 may use a reference image that has been pre-processed in the same manner as the pre-processing unit 610 as each of one or more reference images stored in the reference image storage unit 145. The pre-processing unit 610 may also be applied to a configuration in which the calculation unit 415 is used instead of the calculation unit 115 in FIG. 1.

Figure 7 shows the configuration of a determination device 700 according to a third modified example of this embodiment. The determination device 700 according to this modified example is a modification of the determination device 10 in Figure 1, modified so as to perform a determination that takes into account the cell culture conditions and imaging conditions. In Figure 6, the components with the same reference numerals as in Figure 1 are the same as those in Figure 1, except that at least one of the culture conditions or imaging conditions is used, and therefore will not be described below except for the differences.

The determination device 700 further includes a culture unit 710. The culture unit 710 inputs cell culture conditions and cultures the cells in the measurement target 20 according to the culture conditions. The cell culture conditions may be represented by a set of at least one of environmental parameters such as temperature, operational parameters such as drug administration during the culture process, or other culture-related parameters.

In this modified example, the imaging unit 100 inputs the imaging conditions of the measurement target 20 and images the measurement target 20 according to the imaging conditions. The imaging conditions may be represented by a set of at least one of the following parameters that may cause differences in the captured image even when the same subject is imaged: lens magnification, exposure time, illumination, imaging position (X, Y, Z coordinates, etc.), or other parameters.

The photographed image storage unit 105 stores the photographed image in association with the culture conditions and the photographing conditions. The acquisition unit 110 acquires the culture conditions and the photographing conditions related to the photographed image to be processed in addition to the photographed image. The calculation unit 115 calculates the degree of abnormality based on one or more reference images of cells normally cultured under culture conditions corresponding to the acquired culture conditions and photographed under the photographing conditions corresponding to the acquired photographing conditions. In this case, the model storage unit 155 stores multiple reconstructed models for each culture condition and photographing condition, and the calculation unit 115 may reconstruct the photographed image using a reconstructed model corresponding to the culture conditions and the photographing conditions related to the photographed image to be processed, among the multiple reconstructed models stored in the model storage unit 155. In addition, the abnormality calculation unit 125 may switch the threshold value used to determine whether or not there is an abnormality in the cell culture depending on the culture conditions and the photographing conditions. The learning processing unit 150 may perform a learning process of the reconstructed model using one or more reference images for each culture condition and photographing condition.

According to this modified example, the determination device 700 can perform reconstruction processing and abnormality calculation suitable for the culture conditions and imaging conditions related to the captured images for the captured images obtained under various culture conditions and imaging conditions. This allows the determination device 700 to appropriately determine whether or not there is an abnormality in the cell culture based on the captured images obtained under various culture conditions and imaging conditions.

The determination device 700 may make a determination by taking into account only one of the culture conditions and the imaging conditions. The determination device 700 may also include a calculation unit 415 instead of the calculation unit 115.

Figure 8 shows the configuration of a determination device 800 according to a fourth modified example of this embodiment. The determination device 800 according to this modified example is a modification of the determination device 10 in Figure 1, modified so that it performs determination using images captured at multiple locations. In Figure 8, the components with the same reference numerals as in Figure 1 are the same as those in Figure 1, except that at least one of the culture conditions or the photographing conditions is used, and therefore will not be described below except for the differences.

In this modified example, the acquisition unit 110 acquires a captured image for each of the multiple fields of view. The acquisition unit 110 may acquire partial captured images corresponding to each of the multiple fields of view at once by acquiring a single captured image including the multiple fields of view. Alternatively, the acquisition unit 110 may acquire multiple captured images obtained by capturing each of the multiple fields of view separately.

Here, the acquisition unit 110 may acquire multiple captured images of multiple locations within a culture unit, such as a plate used to culture the measurement target 20 or a well within a plate. Alternatively, the acquisition unit 110 may acquire each captured image by capturing images of multiple plates or multiple wells within the field of view.

In place of the calculation unit 115 in FIG. 1, the determination device 800 includes a distribution unit 810 and multiple calculation units 815-1 to N (also referred to as "calculation units 815"). The distribution unit 810 distributes captured images of multiple fields of view to the multiple calculation units 815-1 to N. In the example shown in this figure, the determination device 800 includes N calculation units 815.

Each of the multiple calculation units 815-1 to N is connected to the distribution unit 810. Each calculation unit 815 calculates the degree of abnormality of the captured image received from the distribution unit 810. Each calculation unit 815 may have a configuration similar to that of the calculation unit 115 in FIG. 1, and may calculate the degree of abnormality of the captured image in the same manner as the calculation unit 115. Alternatively, each calculation unit 815 may have a configuration similar to that of the calculation unit 415 in FIG. 4, and may calculate the degree of abnormality of the captured image in the same manner as the calculation unit 415.

The determination unit 130 is connected to each of the multiple calculation units 815-1 to N. The determination unit 130 outputs a determination result that integrates the degree of abnormality of each of the captured images in the multiple fields of view calculated by each of the multiple calculation units 815.

In this modified example, the model generation unit 137 is connected to the distribution unit 810 and the multiple calculation units 815-1 to N. The learning processing unit 150 in the model generation unit 137 receives the multiple captured images from the calculation unit 815. The learning processing unit 150 receives from each calculation unit 815 the degree of abnormality of the captured image that each calculation unit 815 processes, and adds the captured image to the reference image storage unit 145 as a new reference image, provided that the culture of the cells captured in that captured image is normal. The learning processing unit 150 then performs learning processing of the reconstruction model or feature extraction model using the one or more reference images to which the new reference image has been added. The reconstruction model or feature extraction model generated by the learning processing unit 150 is used by each of the multiple calculation units 815-1 to N.

Figure 9 shows the judgment flow of the judgment device 800 according to the fourth modified example of this embodiment. In the judgment flow of this modified example, the operations of the blocks with the same step numbers as in Figure 2 are the same as the operations of the corresponding blocks in Figure 2, so descriptions will be omitted below except for the differences.

In S200, the imaging unit 100 captures a captured image for each of the multiple fields of view. In S210, the acquisition unit 110 acquires a captured image for each of the multiple fields of view. Here, the acquisition unit 110 may acquire a captured image including multiple fields of view for each culture plate or well. Alternatively, the acquisition unit 110 may acquire a captured image including each of the multiple fields of view for each of the multiple culture plates or wells.

In S915, the distribution unit 810 distributes the captured images of the multiple fields of view acquired by the acquisition unit 110 to the multiple calculation units 815. Here, when a single captured image including multiple fields of view is received, the distribution unit 810 cuts out partial captured images for each of the multiple fields of view from the received captured image and distributes them to each of the multiple calculation units 815-1 to N. When an individual captured image is received for each of the multiple fields of view, the distribution unit 810 distributes each of the multiple captured images to each of the multiple calculation units 815-1 to N.

Between S917 corresponding to the beginning of the loop and S945 corresponding to the end of the loop, the calculation units 815-1 to N perform a loop process of executing the process between S917 and S945 for each captured image distributed by the distribution unit 810. In this modification, in S940 and S270, the learning processing unit 150 in the model generation unit 137 adds a captured image that is determined to have been normal in cell culture using the degree of abnormality ("Y" in S940) as a new reference image to the reference image storage unit 145 (S270). That is, in this modification, when the learning processing unit 150 determines that the cell culture was normal in some of the captured images and that there was an abnormality in the cell culture in the remaining captured images, the learning processing unit 150 uses the captured images that were determined to have been normal in cell culture individually as new reference images, even if the determination unit 130 determines that there was an abnormality in the cell culture as a whole.

In S947, the determination unit 130 outputs a determination result that integrates the degree of abnormality of each captured image of the multiple fields of view calculated by each of the multiple calculation units 815. For example, when the determination unit 130 uses a degree of abnormality that increases the more abnormal the cell culture is, the determination unit 130 outputs a determination result that there is an abnormality in the cell culture when the degree of abnormality of at least one captured image exceeds a threshold. Alternatively, the determination unit 130 may output a determination result that there is an abnormality in the cell culture when the degree of abnormality of more than a predetermined number of captured images exceeds a threshold.

When a determination result that there was an abnormality in the cell culture is output ("N" in S240), the analysis processing unit 132 generates an analysis image in S250 that indicates which area in each captured image contributed to the determination that there was an abnormality in the cell culture. In S260, the result output unit 135 outputs the integrated determination result by the determination unit 130. Furthermore, when it is determined that there was an abnormality in the cell culture, the result output unit 135 outputs each analysis image generated by the analysis processing unit 132 and corresponding to each captured image.

The determination device 800 described above can comprehensively determine whether or not there is an abnormality in the cell culture using captured images of multiple fields of view, such as for each culture plate or well, or for multiple culture plates or multiple wells. This allows the determination device 800 to make a rational determination using captured images of multiple fields of view, even when there is a statistically certain probability that an abnormality will occur in the culture of some cells.

Figure 10 shows the configuration of a determination device 1000 according to a fifth modified example of this embodiment. The determination device 800 according to this modified example is a modification of the determination device 10 in Figure 1, modified so that it performs determination using multiple types of models. In Figure 10, the components with the same reference numerals as in Figure 1 are basically the same as in Figure 1, and therefore will not be described below except for the differences.

The determination device 1000 includes multiple calculation units 1015-1 to M (also referred to as "calculation units 1015") instead of the calculation unit 115 in FIG. 1. In the example shown in this figure, the determination device 1000 includes M calculation units 1015.

The multiple calculation units 1015-1 to 1015-M are connected to the acquisition unit 110. Each calculation unit 1015 receives the same captured image acquired by the acquisition unit 110. Then, each calculation unit 1015 calculates the degree of abnormality of the captured image using the model provided to each calculation unit 1015.

Each calculation unit 1015 may have a configuration similar to that of the calculation unit 115 in FIG. 1. In this case, the multiple reconstructed image generation units 120 included in the multiple calculation units 1015 input the captured image to multiple different reconstruction models, respectively, to generate multiple reconstructed images. The multiple abnormality calculation units 125 included in the multiple calculation units 1015 calculate the abnormality degree for each of the multiple reconstructed images.

Each calculation unit 1015 may have a configuration similar to that of the calculation unit 415 in FIG. 4. In this case, the multiple feature amount calculation units 420 of the multiple calculation units 1015 input the captured image to each of multiple feature extraction models different from each other to calculate multiple feature amounts. Each of the multiple abnormality degree calculation units 425 of the multiple calculation units 1015 calculates the abnormality degree based on the feature amount calculated from the captured image using the different feature extraction models and the reference feature amount of one or more reference images calculated using the feature extraction models. Note that some of the multiple calculation units 1015 may have a configuration similar to that of the calculation unit 115 in FIG. 1, and the other part may have a configuration similar to that of the calculation unit 415 in FIG. 4.

The judgment unit 130 is connected to each of the multiple calculation units 1015-1 to M. The judgment unit 130 outputs a judgment result that integrates the degree of abnormality for each model calculated by each of the multiple calculation units 1015.

In addition, the determination device 1000 may include multiple model generation units 1037-1 to M (also referred to as "determination devices 1037") instead of the model generation unit 137 in FIG. 1. Each determination device 1037 adds the captured image to the reference image storage unit 145 in each determination device 1037 as a new reference image, provided that the culture of the cells captured in that captured image is determined to be normal. Then, the learning processing unit 150 in each determination device 1037 performs learning processing of the reconstruction model or feature extraction model using the one or more reference images to which the new reference image has been added. In this modified example, the multiple models generated by the multiple determination devices 1037 are used by the multiple calculation units 1015-1 to M, respectively.

Here, the multiple reconstruction models or feature extraction models may differ from each other by using different machine learning algorithms, such as a model using an autoencoder, a model using a generative adversarial network, or a machine learning model other than a neural network, such as an SVM or a statistical learning model. The multiple reconstruction models or feature extraction models may differ from each other by using different machine learning parameters, such as the number of layers in a neural network or the number of neurons per layer. The multiple reconstruction models or feature extraction models may differ in the way they learn, such as using different sets of training data (e.g., sets of reference images used for learning) or having different learning intensities.

Figure 11 shows the judgment flow of the judgment device 1000 according to the fifth modified example of this embodiment. In the judgment flow of this modified example, the operations of the blocks with the same step numbers as in Figure 2 are the same as the operations of the corresponding blocks in Figure 2, so descriptions will be omitted below except for the differences.

In S200, the imaging unit 100 captures an image of a field of view including cells, and stores the captured image in the captured image storage unit 105. In S210, the acquisition unit 110 acquires the captured image from the captured image storage unit 105.

The determination device 1000 performs a loop process that executes each process between S1117 corresponding to the beginning of the loop and S1145 corresponding to the end of the loop for each of a plurality of different feature extraction models. In the loop process, the plurality of calculation units 1015-1 to 1015-M executes the same processes as S220 and S230 in FIG. 2 or S520 and S530 in FIG. 5.

In S1147, the determination unit 130 outputs a determination result that integrates the multiple degrees of abnormality output by the multiple calculation units 1015. The determination unit 130 may calculate a determination result as to whether or not there has been an abnormality in the cell culture using statistics of the multiple degrees of abnormality. The determination unit 130 may use at least one of the mean, median, maximum, minimum, variance, or other values as such a statistical amount. For example, the determination unit 130 determines that there has been an abnormality in the cell culture when the average value of the multiple degrees of abnormality exceeds a predetermined threshold value.

The judgment unit 130 may use the degree of abnormality output by each calculation unit 1015 to calculate the individual judgment result by each calculation unit 1015, and determine the final judgment result by majority vote or the like. For example, when the judgment unit 130 uses the degree of abnormality that becomes large in response to an abnormality in the cell culture, if the degree of abnormality output by each calculation unit 1015 exceeds a predetermined threshold, the judgment unit 130 judges that there is an abnormality in the cell culture as a result of the calculation unit 1015 alone. Then, the judgment unit 130 determines the judgment result that there is an abnormality in the cell culture by taking a majority vote of the judgment results based on the degree of abnormality of each calculation unit 1015, or by a condition that the abnormality in the cell culture is judged to be abnormal for a degree of abnormality exceeding a predetermined number.

The determination device 1000 performs the processes of S240 to S270 in FIG. 11 in the same manner as S240 to S270 in FIG. 2. Here, in S250, the analysis processing unit 132 may generate an analysis image that displays the distribution of the degree of influence on the degree of abnormality output by each calculation unit 1015 on the captured image. For example, the analysis processing unit 132 may generate an analysis image on the captured image by combining the difference between the reconstructed image generated by the reconstructed image generating unit 120 in each calculation unit 1015 and the captured image. In this case, the analysis processing unit 132 may reflect the difference between the reconstructed image and the captured image in the analysis image on the condition that the degree of abnormality output by the calculation unit 1015 exceeds a predetermined threshold. In this way, the analysis processing unit 132 can indicate the area in the captured image that has influenced the degree of abnormality when it should be determined that there is an abnormality in the cell culture based on the degree of abnormality of the calculation unit 1015 alone.

According to the determination device 1000 described above, it is possible to determine whether or not there is an abnormality in the cell culture by using multiple models. This allows the determination device 1000 to further improve the accuracy of the determination.

Note that some models for analyzing captured images provide probabilistic output. When at least one of the multiple calculation units 1015 uses such a probabilistic model, the determination device 1000 may perform multiple attempts to evaluate the captured image using the calculation unit 1015 that uses the probabilistic model, and use the resulting statistics of the degree of abnormality for multiple attempts to determine the determination result.

Various embodiments of the present invention may be described with reference to flow charts and block diagrams, where a block may represent (1) a stage of a process in which an operation is performed or (2) a section of an apparatus responsible for performing an operation. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and/or a processor provided with computer readable instructions stored on a computer readable medium. Dedicated circuitry may include digital and/or analog hardware circuitry and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, memory elements such as flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

A computer-readable medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor or programmable circuitry of a programmable data processing apparatus, such as a general-purpose computer, special-purpose computer, or other computer, either locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc., to execute the computer-readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

12 shows an example of a computer 2200 in which aspects of the present invention may be embodied in whole or in part. A program installed on the computer 2200 may cause the computer 2200 to function as or perform operations associated with an apparatus or one or more sections of the apparatus according to an embodiment of the present invention, and/or to perform a process or steps of the process according to an embodiment of the present invention. Such a program may be executed by the CPU 2212 to cause the computer 2200 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphics controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input/output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input/output controller 2220. The computer also includes legacy input/output units such as a ROM 2230 and a keyboard 2242, which are connected to the input/output controller 2220 via an input/output chip 2240.

The CPU 2212 operates according to the programs stored in the ROM 2230 and the RAM 2214, thereby controlling each unit. The graphics controller 2216 retrieves image data generated by the CPU 2212 into a frame buffer or the like provided in the RAM 2214 or into itself, and causes the image data to be displayed on the display device 2218.

The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads programs or data from the DVD-ROM 2201 and provides the programs or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 2230 stores therein a boot program, etc., executed by computer 2200 upon activation, and/or a program that depends on the hardware of computer 2200. Input/output chip 2240 may also connect various input/output units to input/output controller 2220 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

The programs are provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The programs are read from the computer-readable medium, installed in the hard disk drive 2224, RAM 2214, or ROM 2230, which are also examples of computer-readable media, and executed by the CPU 2212. The information processing described in these programs is read by the computer 2200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the manipulation or processing of information according to the use of the computer 2200.

For example, when communication is performed between computer 2200 and an external device, CPU 2212 may execute a communication program loaded into RAM 2214 and instruct communication interface 2222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 2212, communication interface 2222 reads transmission data stored in a transmission buffer processing area provided in RAM 2214, hard disk drive 2224, DVD-ROM 2201, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

The CPU 2212 may also cause all or a necessary portion of a file or database stored on an external recording medium such as the hard disk drive 2224, the DVD-ROM drive 2226 (DVD-ROM 2201), an IC card, etc. to be read into the RAM 2214, and perform various types of processing on the data on the RAM 2214. The CPU 2212 then writes back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. CPU 2212 may perform various types of processing on data read from RAM 2214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to RAM 2214. CPU 2212 may also search for information in a file, database, etc. in the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, CPU 2212 may search for an entry that matches a condition, in which an attribute value of the first attribute is specified, from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described program or software module may be stored on a computer-readable medium on or near the computer 2200. Also, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 2200 via the network.

The present invention has been described above using an embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiment. It is clear from the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in this order.

10 Determination device, 20 Measurement object, 100 Imaging unit, 105 Photographed image storage unit, 110 Acquisition unit, 115 Calculation unit, 120 Reconstruction image generation unit, 125 Abnormality calculation unit, 130 Determination unit, 132 Analysis processing unit, 135 Result output unit, 137 Model generation unit, 145 Reference image storage unit, 150 Learning processing unit, 155 Model storage unit, 415 Calculation unit, 420 Feature calculation unit, 425 Abnormality calculation unit, 600 Determination device, 610 Preprocessing unit, 700 Determination device, 710 Culture unit, 800 Determination device, 810 Distribution unit, 815-1 to N Calculation unit, 1000 Determination device, 1015-1 to M Calculation unit, 1037-1 to M Model generation unit, 2200 Computer, 2201 DVD-ROM, 2210 host controller, 2212 CPU, 2214 RAM, 2216 graphic controller, 2218 display device, 2220 input/output controller, 2222 communication interface, 2224 hard disk drive, 2226 DVD-ROM drive, 2230 ROM, 2240 input/output chip, 2242 keyboard

Claims (19)

an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
The calculation unit is
a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images;
an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image,
the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image;
The determination device further comprises a learning processing unit that performs learning processing of the reconstructed model using the one or more reference images obtained by capturing a field of view including cells that have been determined to have been normally cultured. The determination device according to claim 1, wherein the learning processing unit adds the captured image for which the determination unit has not determined that there was an abnormality in cell culture to the set of one or more reference images as a new reference image, and performs a learning process for the reconstruction model. an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
The calculation unit is
a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images;
an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image,
the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image;
A plurality of the calculation units are provided,
The plurality of reconstructed image generating units included in the plurality of calculation units input the captured image to each of the plurality of reconstructed models different from each other to generate a plurality of reconstructed images;
The plurality of abnormality degree calculation units included in the plurality of calculation units calculate the abnormality degree for each of the plurality of reconstructed images,
The determination unit outputs a determination result obtained by integrating the degrees of abnormality of each of the plurality of reconstructed images. an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
The acquisition unit acquires the captured image for each of a plurality of fields of view,
A plurality of the calculation units are provided,
The plurality of calculation units calculate the degree of abnormality of the captured image of each of the plurality of fields of view,
The determination unit outputs a determination result that integrates the degrees of abnormality of the captured images in the plurality of fields of view. The determination device according to claim 4 , wherein the determination unit outputs a determination result that integrates the degree of abnormality for each culture plate or each well. The determination device according to claim 4 , wherein the determination unit outputs a determination result that integrates the degrees of abnormality for a plurality of culture plates or wells. an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determination unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
The acquisition unit further acquires cell culture conditions,
The calculation unit calculates the degree of abnormality based on the one or more reference images obtained by photographing cells normally cultured under culture conditions corresponding to the acquired culture conditions. The calculation unit is
A feature amount calculation unit that calculates a feature amount of the captured image;
The determination device according to claim 1 , further comprising: an abnormality degree calculation unit that calculates the abnormality degree based on a feature amount of the captured image and each of the reference feature amounts of the one or more reference images. The acquisition unit acquires the captured images of a cell culture process in time series,
The determination device according to claim 1 , wherein the calculation unit calculates the degree of abnormality based on the one or more reference images obtained by capturing a normal cell culture process in time series. A pre-processing unit that performs pre-processing on the captured image is further provided,
The determination device according to claim 1 , wherein the calculation unit calculates the degree of abnormality by using the captured image that has been preprocessed. The determination device according to claim 10 , wherein the pre-processing includes at least one of a three-dimensional reconstruction process and an edge enhancement process of an image of a cell or a cell cluster. The determination device acquires a captured image of a field of view including cells;
The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
Calculating the degree of abnormality of the captured image
The determination device generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images;
The determination device calculates the degree of abnormality based on a difference between the captured image and the reconstructed image,
In generating the reconstructed image, the determination device extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model that has been trained to restore each reference image from the extracted features, thereby generating the reconstructed image;
The determination method further includes performing a learning process for the reconstruction model by the determination device using the one or more reference images obtained by capturing a field of view including cells determined to have been normally cultured. The determination device acquires a captured image of a field of view including cells;
The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
Calculating the degree of abnormality of the captured image
The determination device generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images;
The determination device calculates the degree of abnormality based on a difference between the captured image and the reconstructed image,
In generating the reconstructed images, the determination device extracts features of each of the one or more reference images, and inputs the captured images into a plurality of different reconstruction models that are trained to restore each of the reference images from the extracted features, thereby generating a plurality of the reconstructed images;
The determination method, in determining whether or not an abnormality has occurred in the cell culture, wherein the determination device outputs a determination result that integrates the degree of abnormality of each of the plurality of reconstructed images. The determination device acquires a captured image of a field of view including cells;
The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
In acquiring the captured images, the determination device acquires the captured images for each of a plurality of fields of view,
In calculating the degree of abnormality of the captured image, the determination device calculates the degree of abnormality of each of the captured images in the plurality of fields of view,
The method of determining whether or not there is an abnormality in the cell culture, the determination device outputs a determination result that integrates the degree of abnormality of the captured images of each of the multiple fields of view. The determination device acquires a captured image of a field of view including cells;
The determination device calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
The determination device determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality,
The determination device further acquires cell culture conditions,
A determination method in which, in calculating the degree of abnormality of the captured image, the determination device calculates the degree of abnormality based on one or more reference images photographing cells normally cultured under culture conditions corresponding to the acquired culture conditions. The method is executed by a computer, causing the computer to:
an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determining unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality;
The calculation unit is
a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images;
an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image,
the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image;
The computer,
a learning processing unit that performs learning processing of the reconstructed model using the one or more reference images obtained by capturing a field of view including cells that have been determined to have been normally cultured. The method is executed by a computer, causing the computer to:
an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determining unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality;
The calculation unit is
a reconstructed image generating unit that generates a reconstructed image by reconstructing the captured image so as to be similar to the one or more reference images;
an abnormality degree calculation unit that calculates the abnormality degree based on a difference between the captured image and the reconstructed image,
the reconstructed image generating unit extracts features of each of the one or more reference images, and inputs the captured image into a reconstruction model trained to restore each of the reference images from the extracted features, thereby generating the reconstructed image;
causing the computer to function as a plurality of the calculation units;
The plurality of reconstructed image generating units included in the plurality of calculation units input the captured image to each of the plurality of reconstructed models different from each other to generate a plurality of reconstructed images;
The plurality of abnormality degree calculation units included in the plurality of calculation units calculate the abnormality degree for each of the plurality of reconstructed images,
The determination unit outputs a determination result obtained by integrating the degrees of abnormality of each of the plurality of reconstructed images. The method is executed by a computer, causing the computer to:
an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determining unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality;
The acquisition unit acquires the captured image for each of a plurality of fields of view,
causing the computer to function as a plurality of the calculation units;
The plurality of calculation units calculate the degree of abnormality of the captured image of each of the plurality of fields of view,
The determination unit outputs a determination result that integrates the degrees of abnormality of the captured images of the plurality of fields of view. The method is executed by a computer, causing the computer to:
an acquisition unit for acquiring a captured image of a field of view including cells;
A calculation unit that calculates a degree of abnormality of the captured image based on one or more reference images of normally cultured cells;
a determining unit that determines whether or not there is an abnormality in the culture of the cells captured in the captured image based on the degree of abnormality;
The acquisition unit further acquires cell culture conditions,
The calculation unit calculates the degree of abnormality based on the one or more reference images obtained by photographing cells normally cultured under culture conditions corresponding to the acquired culture conditions.

Images