JP7601982B2 - Character recognition device and image preprocessing method - Google Patents

Description

The present invention relates to a character recognition device and an image preprocessing method.

Optical character recognition (OCR) technology is used to convert images of handwritten or printed characters optically read into digital data (e.g., character codes) that can be used by a computer. Optical reading is achieved by optical devices such as image scanners and digital cameras. The read image is then converted into digital data through image processing such as pattern recognition.

Also, so-called artificial intelligence (AI) technology has been developing remarkably. Important milestones in AI technology in recent years include deep learning, which uses a deep neural network with multiple intermediate layers between the input layer and the output layer, and the Transformer, an encoder/decoder type model that uses an attention mechanism.

One of the main application areas of AI technology is image processing technology. As mentioned above, image processing is used in OCR technology, and AI-OCR technology, which applies AI technology to OCR technology, is currently at the beginning of development (for example, Patent Document 1).

JP 2023-003648 A

Distortion correction using AI technology in OCR processing is still a developing field, and various technological developments are expected in the future. The smaller the distortion of the image input to OCR processing (hereinafter referred to as the actual input image), the more appropriate character recognition will be achieved. A standout technology or combination of technologies that can become the de facto standard for AI-OCR has yet to be found.

In view of the above, the present invention aims to appropriately process characters to be recognized that are written in an actual input image.

To achieve the above object, the character recognition device according to the present invention includes a feature point detection unit that detects a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image, an outlier point removal unit that removes outliers from the plurality of feature points detected by the feature point detection unit, a distortion correction unit that corrects distortion of the actual input image using the removed feature points after the outlier point removal unit has removed the outliers, and a character recognition unit that performs character recognition on corrected characters included in areas corresponding to the recognition areas in the corrected actual input image after correction by the distortion correction unit.

The image preprocessing method according to the present invention includes detecting, by a computer processor, a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image, removing outliers from the detected plurality of feature points, and correcting distortion of the actual input image using the removed feature points after the outliers have been removed.

The above configuration makes it possible to appropriately process characters to be recognized that are written in an actual input image. Note that the above configuration may provide other effects instead of or in addition to the above effect.

1 is a diagram illustrating a character recognition system S including a character recognition device 20 according to a first embodiment. FIG. 2 is a hardware configuration diagram of the user terminal 10 according to the first embodiment. 1 is a hardware configuration diagram of a character recognition device 20 according to a first embodiment. FIG. 2 is a diagram illustrating a software configuration of the character recognition device 20 according to the first embodiment. 5 is a flowchart showing detailed processing of character recognition according to the first embodiment. FIG. 2 is a diagram showing an example of a template image _IT according to the first embodiment. FIG. 2 is a diagram showing an example of an actual input image I _R according to the first embodiment. FIG. 4 is an explanatory diagram of feature point detection according to the first embodiment. FIG. 2 is a diagram showing an example of a corrected actual input image _IRA according to the first embodiment. FIG. 10 is a diagram showing a comparative example of FIG. 9 . FIG. 11 is a diagram illustrating a software configuration of a character recognition device 20 according to a second embodiment.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the attached drawings. In this specification and drawings, elements that can be described in the same way may be designated by the same reference numerals to avoid redundant description.

Each of the embodiments described below is merely one example of a configuration that can realize the present invention. Each of the following embodiments can be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions. Not all of the combinations of elements included in each of the following embodiments are necessarily essential to realize the present invention, and some of the elements can be omitted as appropriate. Therefore, the scope of the present invention is not limited by the configurations described in each of the following embodiments. As long as there are no mutual contradictions, a configuration that combines multiple configurations described in the following embodiments can also be adopted.

1. First embodiment Fig. 1 is a diagram illustrating a character recognition system S including a character recognition device 20 according to a first embodiment. The character recognition system S includes a user terminal 10 and the character recognition device 20. The character recognition system S may include other components. The other components are, for example, a front-end server that is disposed between the user terminal 10 and the character recognition device 20 and executes various data processing.

The user terminal 10 is a terminal device such as a smartphone or a PC used by a user. The user uses the user terminal 10 to use applications and services provided by the character recognition device 20. The user terminal 10 is connected to the character recognition device 20 via, for example, a wireless communication network and the Internet.

The character recognition device 20 is a server device that performs OCR processing on an image (actual input image) sent from the user terminal 10. The character recognition device 20 performs OCR processing on actual input images corresponding to various standard forms, such as images of medical certificates for insurance claims and images of application forms for opening a bank account.

The character recognition device 20 may transmit the character recognition result by the OCR process to another server device. The character recognition device 20 may also be a server device that executes various application processes including the OCR process. That is, the character recognition device 20 may provide only the functions described in this embodiment, or may also provide other functions. The character recognition device 20 may be arranged in an on-premise environment, or in a cloud environment provided by another company. The character recognition device 20 may also be configured to execute only normal OCR processing by the character recognition unit 28, and to separately arrange a pre-processing device that is a computer including the feature point detection unit 22, the outlier removal unit 24, and the distortion correction unit 26, and to obtain a character recognition result by OCR processing an image received from a user terminal and processed by this pre-processing device in the character recognition unit 28 of the character recognition device 20. Furthermore, all or part of the parts of the character recognition device 20 may be configured as one application, which is installed and executed on the user terminal.

The character recognition device 20 includes, as functional units for implementing OCR processing, a feature point detection unit 22, an outlier removal unit 24, a distortion correction unit 26, and a character recognition unit 28. Details of each unit will be described later.

Figure 2 is a hardware configuration diagram of the user terminal 10 according to the first embodiment. As shown in Figure 2, the user terminal 10 has a processor 101, a memory 102, an input/output interface 103, and a communication interface 104. The above elements provided in the user terminal 10 are connected to each other by an internal bus. Note that the user terminal 10 may have hardware elements other than the elements shown in Figure 2.

The processor 101 is a computing element that realizes various functions of the user terminal 10. The processor 101 may be a system-on-a-chip (SoC) that includes elements such as a central processing unit (CPU), a graphics processing unit (GPU), and a memory controller.

The memory 102 is composed of a storage medium such as a RAM (Random Access Memory) or an eMMC (embedded Multi Media Card). The memory 102 is an element that temporarily or permanently stores programs and data used to execute various processes in the user terminal 10. The programs include one or more instructions for the operation of the user terminal 10. The processor 101 realizes the functions of the user terminal 10 by expanding and executing the programs stored in the memory 102 in the memory 102 and/or a system memory (not shown).

The input/output interface 103 is an interface that accepts operations on the user terminal 10 and supplies them to the processor 101, as well as presenting various information to the user, and is, for example, a touch panel or a keyboard and display.

The communication interface 104 is a circuit that performs various signal processing to realize Internet communication, such as a network interface card (NIC).

Fig. 3 is a hardware configuration diagram of the character recognition device 20 according to the first embodiment. As shown in Fig. 3, the character recognition device 20 has a processor 201, a memory 202, an input/output interface 203, and a communication interface 204. The above elements provided in the character recognition device 20 are connected to each other by an internal bus. Note that the character recognition device 20 may have hardware elements other than the elements shown in Fig. 3.

The processor 201 is a computing element that realizes various functions of the character recognition device 20. The processor 201 may be a CPU, and may further include other processors such as a GPU.

The memory 202 is composed of a storage medium such as a RAM, a ROM (Read Only Memory), a HDD (Hard Disk Drive), or an SSD (Solid State Drive). The memory 202 is an element that temporarily or permanently stores programs and data used to execute various processes in the character recognition device 20. The programs include one or more instructions for the operation of the character recognition device 20. The processor 201 realizes the functions of the character recognition device 20 by expanding and executing the programs stored in the memory 202 in the memory 202 and/or a system memory (not shown).

The input/output interface 203 is an interface that accepts operations on the character recognition device 20 and supplies them to the processor 201, and presents various information to the user, and is, for example, a keyboard and a display. Note that the character recognition device 20 may not have the input/output interface 203 and may be remotely operated.

The communication interface 204 is a circuit that performs various signal processing to realize Internet communication, such as a network interface card (NIC).

Figure 4 is a software configuration diagram of the character recognition device 20 according to the first embodiment. As shown in Figure 4, the character recognition device 20 has a control unit 210, a memory unit 220, and a communication unit 230.

The control unit 210 is a software element that realizes various functions including the feature point detection unit 22, the outlier removal unit 24, the distortion correction unit 26, and the character recognition unit 28, and is realized by the processor 201 described above. The operation of the control unit 210 will be briefly described below.

The feature point detection unit 22 detects a plurality of corresponding feature points PF between a template image _IT including a plurality of recognition areas RA and an actual input image _IR in which characters to be recognized are written in a format _F corresponding to the template image _IT .

The outlier removal section 24 removes the outlier points P _O from the plurality of feature points P _F detected by the feature point detection section 22 .

The distortion correction unit 26 corrects the distortion of the actual input image I _R by using the removed feature points P _FE obtained after the outlier removal unit 24 has removed the outlier points.

The character recognition unit 28 performs character recognition on the corrected character C′ included in the area CA, which corresponds to the recognition area RA in the corrected actual input image _IRA corrected by the distortion correction unit 26 .

The storage unit 220 is an element that stores various data and programs used by the control unit 210, and is realized by the memory 202 that cooperates with the processor 201.

The communication unit 230 is an element that communicates with other devices under the control of the control unit 210, and is realized by the communication interface 204 that cooperates with the processor 201.

Details of the character recognition process according to the first embodiment will be described with reference to Figures 5 to 10. Figure 5 is a flowchart showing the details of the character recognition process according to the first embodiment.

In step S510, first, the feature point detection unit 22 receives the actual input image I _R transmitted from the user terminal 10 and reads out the template image I _T stored in the storage unit 220.

6 is a diagram showing an example of a template image _IT according to the first embodiment. The template image _IT is image data used to print a paper medium (format F) on which a user writes characters. The format F may be provided to the user as a printed paper medium, or may be printed by the user himself.

As shown in Fig. 6, the template image _IT includes a plurality of recognition areas RA. The recognition areas RA include, for example, an area in which Chinese characters or numbers are written, and an area to be checked with a check mark. In Fig. 6, for the sake of simplicity, only some of the recognition areas RA are labeled with symbols. The recognition areas RA occupy a predetermined area (e.g., a rectangular area) specified by coordinates in the image in the image data representing the template image _IT , and also occupy a predetermined physical area in the printed format F.

Fig. 7 is a diagram showing an example of a real input image I _R according to the first embodiment. As shown in Fig. 7, the real input image I _R is an image obtained by photographing a format F corresponding to the template image I _T , and includes characters written by a user in the format F. The real input image I _R is image data acquired by photographing the format F, which is an actual paper medium. Therefore, as shown in Fig. 7, the real input image I _R often has distortions caused by multiple factors such as folds in the paper and the photographing angle.

In step S510, the feature point detection unit 22 executes the following feature point detection as preprocessing for the distortion correction of the actual input image I _R by the distortion correction unit 26 (step S530).

8 is an explanatory diagram of feature point detection according to the first embodiment. The feature point detection unit 22 detects a plurality of corresponding feature points _PF between the template image I _T and the actual input image I _R based on, for example, a Local Feature Matching with Transformers (LoFTR) algorithm.

The feature points P _F are characteristic points on an image, such as the boundaries of an area or the corners of a rectangular frame, and in this embodiment correspond to each other between the template image I _T and the actual input image I _R. The corresponding feature points P _F on the template image I _T and the corresponding feature points P _F on the actual input image I _R have similar feature amounts.

The feature point detection unit 22 preferably extracts points having a confidence level (Confidence) indicating the degree of correspondence between the points included in the template image I _T and the points included in the real input image I _R that exceeds a predetermined threshold (e.g., 98% or 99%) as a plurality of feature points P _F. The extraction of the feature points P _{F described above is performed, for example, according to the LoFTR algorithm. The LoFTR algorithm is realized by a trained model trained by supervised learning using teacher data. In the LoFTR algorithm, a plurality of feature points P F corresponding} to each other between the template image I _T and the real input image I _R are searched for, and the searched feature points P _F are pruned according to the confidence level.

In the LoFTR algorithm, first, feature quantities F _T and F _R for each image I _T and I _R are calculated using a convolutional neural network and a transformer. Then, a rough correspondence between the images I _T and I _R is calculated using a transport optimal algorithm, and a detailed correspondence is calculated for small patches. As a result, pixel-level matching is performed between the images I _T and I _R , and multiple corresponding feature points P _F are extracted. As can be seen from the above, the LoFTR algorithm is a method that is applied to the entirety of each image I _T and I _R.

8 is an explanatory diagram of a plurality of feature points P _F detected by the LoFTR algorithm. In FIG. 8, corresponding feature points P _F between the template image I _T and the real input image I _R are indicated by line segments.

The feature point detection unit 22 may detect a plurality of feature points P _F based on an algorithm other than LoFTR, such as SuperPoint or Patch2Pix. That is, the feature point detection unit 22 may execute the process of step S510 on the template image I _T and the actual input image I _R using any detection algorithm.

In step S520, as preprocessing for the distortion correction of the actual input image I _R by the distortion correction unit 26 (step S530), the outlier removal unit 24 removes the outlier points P _O from the plurality of feature points P _F detected by the feature point detection unit 22.

More specifically, the outlier removal unit 24 identifies an outlier from coordinate values corresponding to a plurality of feature points P _F based on a CONSAC (Conditional Sample Consensus) algorithm, and removes an outlier P _O , which is a feature point P _F corresponding to the identified outlier, from feature points P _F to be used for distortion correction in step S530 by the distortion correction unit 26. For example, when the distance between the coordinate values of corresponding feature points P _F exceeds a predetermined threshold, the outlier removal unit 24 may recognize the coordinate values (a set of feature points P _{F )} as an outlier. The predetermined threshold may be set based on a statistical value (average value, median, variance, standard deviation, etc.) of the distance between all corresponding feature points P _{F, or may be set based on a statistical value of the distance between randomly selected partial feature points P F.}

The CONSAC algorithm is realized by a trained model trained by supervised learning and self-supervised learning. In the CONSAC algorithm, samples are updated based on information used when selecting samples from a dataset. Like the LoFTR algorithm, the CONSAC algorithm is a method applied to the entirety of each image I _T and I _R.

In step S530, the distortion correction unit 26 corrects the distortion of the actual input image I _R by using the removed feature points P _FE obtained after the outlier point removal unit 24 has removed the outlier points P _O.

More specifically, the distortion correction unit 26 corrects the real input image I _R based on a Thin Plate Spline algorithm so as to bring the coordinates of the removed feature points P _FE in the real input image I _R closer to the coordinates of the corresponding feature points P _F in the template image I _T , and outputs a corrected real input image I _RA .

The thin plate spline algorithm is an algorithm that uses a set of points in a two-dimensional plane to find a curved surface that passes through the points included in the set. In this embodiment, the thin plate spline algorithm is applied to the entire real input image I _R. Alternatively, as described later, the thin plate spline algorithm may be applied to partial images obtained by dividing the real input image I _R.

Fig. 9 is a diagram showing an example of a corrected actual input image IRA according to the first embodiment, while Fig. 10 is a diagram showing an example of a corrected actual input image _I'RA (i.e., a comparison example of Fig. 9) output from the distortion correction unit 26 when the outlier removal in step _S520 in the flowchart shown in Fig. 5 is not executed.

As shown in FIG. 9, when the above-described steps S510 to S530 are executed on the actual input image I 1 _R , a corrected actual input image I 1 _RA in which the distortion in the format F has been appropriately corrected is acquired.

10, if the outlier removal in step S520 is not performed, a corrected actual input image _I'RA with incomplete distortion correction will be acquired, resulting in a decrease in the accuracy of the subsequent character recognition.

In step S540, the character recognition unit 28 performs character recognition on the corrected character C′ included in the area CA corresponding to the recognition area _RA in the corrected actual input image IRA corrected by the distortion correction unit 26.

According to the above configuration, character recognition is performed on the real input image I _R that has been subjected to distortion correction based on feature point detection and outlier removal, so it is possible to more appropriately recognize characters written in the real input image I _R compared to a configuration that does not do so.

2. Second embodiment Fig. 11 is a software configuration diagram of a character recognition device 20 according to a second embodiment. As shown in Fig. 11, the character recognition device 20 has a control unit 210, a storage unit 220, and a communication unit 230, similar to the first embodiment. Compared to the first embodiment, the control unit 210 of the second embodiment further includes a template selection unit 30 and a correction necessity determination unit 32 as software elements.

The template selection unit 30 uses the LoFTR algorithm and/or the CONSAC algorithm to select a template image I 1 _T to be used by the feature point detection unit 22 from among a plurality of template images I 1 _T.

More specifically, for example, the template selection unit 30 performs feature point detection between the real input image I _R and each of the multiple template images I _T using the feature point detection unit 22, and selects the template image I _T from which the largest number of feature points P _F are detected. Then, steps S510 to S540 of the first embodiment are executed using the selected template image I _T.

Furthermore, the template selection unit 30 may further perform outlier removal on the feature points _PF detected as described above using the outlier removal unit 24, and then select the template image _IT having the largest number of remaining feature points _PF .

According to the above configuration, even if the user of the user terminal 10 does not select a template image _IT , the character recognition device 20 can automatically select the template image _IT to be used.

The correction necessity determination unit 32 uses the LoFTR algorithm and/or the CONSAC algorithm to determine whether or not distortion correction should be performed by the distortion correction unit 26.

More specifically, for example, the correction necessity determination unit 32 performs feature point detection by the feature point detection unit 22 between the actual input image I _R and each of the multiple template images I _T. If the difference in coordinate values between corresponding feature points P _F is large (for example, if the sum of the differences in coordinate values exceeds a predetermined threshold value), it is considered that the distortion of the actual input image I _R is relatively large, and therefore the correction necessity determination unit 32 determines that distortion correction should be performed by the distortion correction unit 26.

Alternatively, if the variance of the differences in coordinate values between corresponding feature points P _F is large (for example, if the variance of the differences in coordinate values exceeds a predetermined threshold value), the distortion of the actual input image I _R is considered to be relatively large, and the correction necessity determination unit 32 determines that distortion correction should be performed by the distortion correction unit 26.

Furthermore, the correction necessity determining section 32 may perform the above-mentioned correction necessity determination after the outlier removing section 24 further performs outlier point removal on the feature points _PF detected as described above.

If it is determined that distortion correction by the distortion corrector 26 is not to be performed, the controller 210 may perform general keystone correction on the actual input image I 1 _R.

According to the above configuration, distortion correction is selectively performed on images with relatively large distortion, i.e., images for which the need for distortion correction is relatively high. Therefore, it is possible to reduce the overall processing load of the character recognition device 20.

In addition, instead of the above-mentioned correction necessity determination, the correction necessity determination unit 32 may determine whether or not distortion correction should be performed by the distortion correction unit 26 based on whether or not the actual input image I _R includes a straight line portion having a predetermined relative length (e.g., 70% or 80% of the total length of the vertical or horizontal side of the format F) or more.

In the first embodiment, in step S530, the distortion correction unit 26 performs distortion correction on the entire real input image I _R. In contrast, in the second embodiment, the distortion correction unit 26 divides the real input image I _R into a plurality of partial images I _P , and then corrects the distortion of the partial images I _P using the removed feature points _PFE included in each partial image I _P. The distortion correction unit 26 may divide the real input image I _R into six partial images I _P , eight partial images I _P , or even more partial images I _P.

The thin plate spline algorithm imposes a significant processing load as the size of the image to be processed increases, and the possibility of problems such as memory leaks also increases. With the above configuration, it is possible to reduce the processing load of distortion correction by the distortion correction unit 26.

In the second embodiment, too, the feature point detection in step S510 and the outlier point removal in step S520 are performed on the entire real input image I _R.

Furthermore, the distortion correction unit 26 may correct the distortion of each partial image IP by using a predetermined number or less of the removed feature points _PFE among the removed feature points _PFE included in the partial image _IP . Furthermore, when selecting the removed feature points _PFE , the distortion correction unit 26 may perform the selection so that the distance between the removed feature _{points PFE} is kept at a predetermined value or more (for example, so that the total distance between the removed feature points _PFE after selection exceeds a predetermined threshold value).

The above configuration makes it possible to further reduce the processing load of distortion correction by the distortion correction unit 26.

In the present embodiment described above, the template selection by the template selection unit 30, the correction necessity determination by the correction necessity determination unit 32, and the distortion correction of the partial image by the distortion correction unit 26 can be performed independently. It will be understood by those skilled in the art that only one or two of the above three independent operations may be performed.

3. Other embodiments 3.1. Modifications Although the embodiments for carrying out the present invention have been described above, the present invention is not limited to the above-mentioned embodiments. It is naturally understood that the above-mentioned embodiments are merely examples and that various modifications are possible. Words, phrases, and other expressions used in the above-mentioned embodiments are merely examples and may be replaced with substantially the same or similar expressions.

After performing the distortion correction described in the above embodiment, the distortion correction unit 26 may use the LoFTR algorithm and/or the CONSAC algorithm to determine whether or not the distortion correction in the corrected actual input image _IRA is appropriate.

The means and/or functions provided by the devices described in the above embodiments can be provided by software recorded in a physical memory device and a computer that executes the software, by software alone, by hardware alone, or by a combination of these. For example, if any of the above devices is provided by electronic circuits that are hardware, it can be provided by digital circuits including a large number of logic circuits, or by analog circuits.

The device described in the above embodiment executes a program stored in a non-transitory tangible storage medium. Execution of this program results in the execution of a method corresponding to the program.

3.2. Supplementary Note A part or all of the above embodiment and modified examples may be described as the following supplementary note, but are not limited to the contents of the following supplementary note. In the following, a relationship is expressed in which a supplementary note that is dependent on multiple supplementary notes is dependent on the supplementary note that is dependent on multiple supplementary notes. All of the dependent relationships of the supplementary notes expressed below are included in the above embodiment.

(Appendix 1)
a feature point detection unit that detects a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
an outlier removal unit that removes outliers from the plurality of feature points detected by the feature point detection unit;
a distortion correction unit that corrects distortion of the actual input image by using removed feature points obtained after the outlier point removal unit has removed the outlier points;
a character recognition unit that performs character recognition on a corrected character included in an area corresponding to the recognition area in the corrected actual input image after correction by the distortion correction unit.

(Appendix 2)
The outlier removal unit identifies an outlier from coordinate values corresponding to the plurality of feature points based on a Conditional Sample Consensus (CONSAC) algorithm, and removes the outlier that is a feature point corresponding to the identified outlier from feature points to be used for distortion correction by the distortion correction unit.

(Appendix 3)
The character recognition device according to claim 2, wherein the distortion correction unit corrects the actual input image based on a Thin Plate Spline algorithm so as to bring coordinates of removed feature points in the actual input image closer to coordinates of corresponding feature points in the template image.

(Appendix 4)
The character recognition device according to claim 2 or 3, wherein the feature point detection unit extracts points whose confidence level, which indicates the degree to which points included in the template image correspond to points included in the actual input image, exceeds a predetermined threshold value as the plurality of feature points.

(Appendix 5)
The character recognition device according to any one of Supplementary Note 2 to Supplementary Note 4, wherein the feature point detection unit detects the plurality of feature points based on a Local Feature Matching with Transformers (LoFTR) algorithm.

(Appendix 6)
The character recognition device according to any one of Supplementary Note 2 to Supplementary Note 5, further comprising a template selection unit that uses the LoFTR algorithm and/or the CONSAC algorithm to select the template image to be used by the feature point detection unit from a plurality of template images.

(Appendix 7)
The character recognition device according to any one of Supplementary Note 2 to Supplementary Note 5, further comprising a correction necessity determination unit that determines whether or not the distortion correction should be performed by the distortion correction unit using the LoFTR algorithm and/or the CONSAC algorithm.

(Appendix 8)
The character recognition device according to any one of Supplementary Note 1 to Supplementary Note 5, further comprising a correction necessity determination unit that determines whether or not the distortion correction should be performed by the distortion correction unit, based on whether or not the actual input image includes a straight line portion having a predetermined relative length or more.

(Appendix 9)
9. The character recognition device according to claim 1, wherein the distortion correction unit divides the actual input image into a plurality of partial images, and then corrects distortion of each partial image by using the removed feature points included in each partial image.

(Appendix 10)
The character recognition device according to claim 9, wherein the distortion correction unit corrects distortion of each partial image by using a predetermined number or less of the removed feature points included in each partial image.

(Appendix 11)
The character recognition device according to claim 10, wherein when selecting the predetermined number or less of removed feature points, the distortion correction unit performs selection such that a distance between removed feature points P is maintained at a predetermined value or more.

(Appendix 12)
a feature point detection unit that detects a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
an outlier removal unit that removes outliers from the plurality of feature points detected by the feature point detection unit;
a distortion correction unit that corrects distortion of the actual input image by using removed feature points obtained after the outlier point removal unit has removed the outlier points.

(Appendix 13)
The computer's processor
Detecting a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
removing outliers from the detected feature points;
and correcting distortion of the real input image using the removed feature points after the outlier points have been removed.

(Appendix 14)
The processor of the character recognition device
Detecting a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
removing outliers from the plurality of feature points;
correcting distortion of the real input image using the removed feature points after the outlier points have been removed;
performing character recognition on a corrected character included in an area corresponding to the recognition area in a corrected actual input image after the correction.

(Appendix 15)
The processor of the character recognition device
Detecting a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
removing outliers from the plurality of feature points;
correcting distortion of the real input image using the removed feature points after the outlier points have been removed;
performing character recognition on a corrected character included in an area corresponding to the recognition area in a corrected actual input image after the correction;

(Appendix 16)
The processor of the character recognition device
Detecting a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
removing outliers from the plurality of feature points;
correcting distortion of the real input image using the removed feature points after the outlier points have been removed;
and performing character recognition on the corrected characters included in an area corresponding to the recognition area in the corrected actual input image after the correction.

REFERENCE SIGNS LIST 10 User terminal 20 Character recognition device 22 Feature point detection unit 24 Outlier removal unit 26 Distortion correction unit 28 Character recognition unit 30 Template selection unit 32 Correction necessity determination unit

Claims (12)

a feature point detection unit that detects a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
an outlier removal unit that identifies an outlier from coordinate values corresponding to the plurality of feature points detected by the feature point detection unit, and removes the outlier that is the feature point corresponding to the identified outlier from feature points to be used for distortion correction;
a distortion correction unit that corrects distortion of the actual input image by using removed feature points obtained after the outlier point removal unit has removed the outlier points;
a character recognition unit that performs character recognition on a corrected character included in an area corresponding to the recognition area in the corrected actual input image after the distortion correction unit has corrected the image,
the feature point detection unit detects the plurality of feature points without performing a process related to distortion correction of the actual input image in advance;
The outlier removal unit removes the outliers based on a CONSAC (Conditional Sample Consensus) algorithm.
Character recognition device. 2. The character recognition device according to claim 1, wherein the outlier removal unit identifies an outlier from a coordinate value of each feature point of the template image detected by the feature point detection unit and a coordinate value of each feature point of the actual input image corresponding to each feature point of the template image, and removes an outlier that is a feature point corresponding to the identified outlier from feature points to be used for distortion correction by the distortion correction unit. 3. The character recognition device according to claim 2, wherein the outlier removal unit recognizes a coordinate value as the outlier when a distance between coordinate values of a pair of corresponding feature points exceeds a threshold value that is set based on a statistical value of distances between all or a part of the corresponding multiple feature points. 4. The character recognition device according to claim 3, wherein the distortion correction unit corrects the actual input image based on a Thin Plate Spline algorithm so as to bring coordinates of removed feature points in the actual input image closer to coordinates of corresponding feature points in the template image. 5. The character recognition device according to claim 4, wherein the feature point detection unit extracts, as the plurality of feature points, points whose degree of certainty, which indicates the degree to which points included in the template image correspond to points included in the actual input image, exceeds a predetermined threshold value. The character recognition device according to claim 5 , wherein the feature point detection unit detects the plurality of feature points based on a Local Feature Matching with Transformers (LoFTR) algorithm. The character recognition device according to claim 6 , further comprising a template selection unit that selects the template image to be used by the feature point detection unit from a plurality of template images by using the LoFTR algorithm and/or the CONSAC algorithm. The character recognition device according to claim 6 , further comprising a correction necessity determination unit that determines whether or not the distortion correction should be performed by the distortion correction unit, using the LoFTR algorithm and/or the CONSAC algorithm. 7. The character recognition device according to claim 1, further comprising a correction necessity determination unit that determines whether or not the distortion correction should be performed by the distortion correction unit, based on whether or not the actual input image includes a straight line portion having a predetermined relative length or more. The character recognition device according to claim 1 , wherein the distortion correction unit divides the actual input image into a plurality of partial images, and then corrects the distortion of each partial image by using the removed feature points included in each partial image. the distortion correction unit corrects distortion of each partial image by using a predetermined number or less of the removed feature points included in each partial image;
The character recognition device according to claim 10 , wherein when the predetermined number or less of removed feature points are selected, the selection is performed such that a distance between the removed feature points is maintained at a predetermined value or more. The computer's processor
Detecting a plurality of corresponding feature points between a template image including a plurality of recognition areas and an actual input image in which characters to be recognized are written in a format corresponding to the template image;
identifying an outlier from coordinate values corresponding to the detected plurality of feature points, and removing the outlier that is the feature point corresponding to the identified outlier from feature points to be used for distortion correction;
and correcting the distortion of the real input image using the removed feature points after the outlier points have been removed;
detecting the plurality of feature points includes detecting the plurality of feature points without performing a process related to distortion correction of the actual input image in advance;
removing the outlier points based on a Conditional Sample Consensus (CONSAC) algorithm;
Image pre-processing methods.

Images