JP7775155B2 - Two-dimensional code reading system and two-dimensional code - Google Patents

Description

The present invention relates to a two-dimensional code reading system and a two-dimensional code.

Indicators that estimate the environment to which a cord has been exposed and management systems that utilize these indicators are known. Patent Document 1 describes a temperature evaluation system that includes "a processing device that includes an input device that acquires image data of an indicator that uses a thermochromic material; a storage device (database) that stores the relationship between color density and temperature for each thermochromic material; a color density estimation unit that estimates the color density of the thermochromic material from the image data; a material identification unit that identifies the thermochromic material used in the indicator; and a temperature estimation unit that selects the relationship between color density and temperature of the thermochromic material identified by the material identification unit from the relationship between color density and temperature for each thermochromic material, and estimates the maximum or minimum temperature that can be reached from the relationship between color density and temperature of the identified thermochromic material and the color density estimated by the color density estimation unit."

This temperature evaluation system is configured with an indicator consisting of a thermochromic material that indicates the environment to which it is exposed using color intensity, and a one-dimensional or two-dimensional barcode. It also obtains color information from a photographed image of the indicator attached to the item, and estimates the item's maximum or minimum temperature. This allows for centralized management of temperature data obtained at each location during distribution by having a management device (e.g., a management server) that manages the environment in which the item is placed, and management terminals (input devices) located at each base.

However, in the system described in Patent Document 1, the shooting environment and shooting equipment have a strong influence when capturing an image of the indicator, making it difficult to accurately obtain color information of the thermochromic material from the captured image of the indicator. Examples of shooting environments that affect images include the type and brightness of the light source. The spectrum of light hitting the subject varies depending on the type of light source (lighting model)—direct sunlight, cloudy daylight, fluorescent lamps, incandescent bulbs—and the degree of light attenuation from the light source. In many cases, there are multiple light sources, and the light from these sources may be mixed. It is not uncommon for the location and orientation of the sensor, as well as the photographer's position, to vary depending on the shooting conditions. Furthermore, the time of shooting (position of the sun) and weather are constantly changing. These factors cause the brightness and spectrum of the light hitting the sensor of the subject to change in various ways.

To address this issue, Patent Document 2 describes a measurement device that includes an analysis means for determining a correction value for the color sample from color image data obtained by simultaneously capturing a color-changing color sensor and three color samples, and a correction means for correcting the color of the color sensor using the correction value. This makes it possible to correct the color of the color sensor using the color samples captured in the same image, even when the color sensor is captured under various light sources.

Patent Document 3 also describes a color evaluation system that obtains highly accurate measurement values from color sensors captured in various environments. The system includes a color evaluation unit that calculates color conversion coefficients between the first and second imaging environments based on the amount of change from the color information of the four or more reference colors in the first imaging environment read from the storage device to the color information of the four or more reference colors in the second imaging environment obtained from the imaging data, and a color evaluation unit that calculates a conversion equation including a term indicating a parallel shift of the affine transformation based on the sensor color obtained from the imaging data during measurement and the color conversion coefficients, thereby correcting the sensor color to appear as if it were captured in the first imaging environment.

Japanese Patent Application Laid-Open No. 2018-205222 JP 2012-093277 A Japanese Patent Application Laid-Open No. 2020-038073

While the technologies described in Patent Documents 2 and 3 can accurately obtain the color tone of the temperature-indicating material, which indicates the environment to which it is exposed through color density, they are subject to system limitations in obtaining the code information within the indicator, the color information of the reference color section, and the color information of the temperature-indicating material section, making it difficult to miniaturize the code and limiting the products for which they can be used. Furthermore, obtaining both code information and highly accurate color information requires slow processing, and reading multiple indicators takes time, resulting in issues that make them unsuitable for practical use.

The present invention aims to solve the above-mentioned problems by providing a two-dimensional code reading system that includes an environmental detection area in the data area that changes color in response to environmental changes, and aims to improve the accuracy and reading speed of obtaining color information from the environmental detection area that changes color in response to environmental changes.

To achieve the above object, the present invention provides a two-dimensional code reading system having an environment detection area in its data area that changes color in response to environmental changes. The system includes an image capture device that captures an image of the two-dimensional code, a data processing device, and a storage device that stores color information for a reference color and position information for the reference color and the environment detection area. The two-dimensional code has a positioning pattern colored with a reference color used to determine the amount of color change depending on the shooting environment. The data processing device recognizes the reference position of the two-dimensional code from the image of the two-dimensional code, and determines the environment detection area and the area colored with the reference color in the acquired image of the two-dimensional code based on the detected reference position and the position information for the reference color and the environment detection area stored in the storage device. The data processing device detects color information for the reference color and the environment detection area, and corrects the color information of the environment detection area using the relationship between the color information for the reference color stored in the storage device and the color information for the detected reference color area. Other aspects of the present invention will be described in the embodiments described below.

This invention provides a two-dimensional code reading system that includes an environmental detection area in the data area that changes color in response to environmental changes, improving the accuracy of obtaining color information from the environmental detection area and the reading speed.

3 is a schematic diagram of a label item and a first two-dimensional code according to the present embodiment. FIG. 3 is a schematic diagram of a finder pattern of a two-dimensional code according to the present embodiment. FIG. 3A and 3B are schematic diagrams of alignment patterns of a two-dimensional code according to the present embodiment. FIG. 4 is a schematic diagram of a second two-dimensional code according to the present embodiment. FIG. 10 is a schematic diagram of a third two-dimensional code according to the present embodiment. FIG. 10 is a schematic diagram of a fourth two-dimensional code according to the present embodiment. FIG. 10 is a schematic diagram of a fifth two-dimensional code according to the present embodiment. FIG. 10 is a schematic diagram of a sixth two-dimensional code according to the present embodiment. FIG. 10 is a schematic diagram of a seventh two-dimensional code according to the present embodiment. FIG. 10 is a schematic diagram of an eighth two-dimensional code according to the present embodiment. FIG. 10 is a schematic diagram of a ninth two-dimensional code according to the present embodiment. FIG. 12 is a schematic diagram of a tenth two-dimensional code according to the present embodiment. FIG. 1 is a diagram illustrating a configuration of a reading system according to an embodiment of the present invention. 10A to 10C are diagrams illustrating a color correction method according to the present embodiment. 10A and 10B are diagrams illustrating a color determination method according to the present embodiment. 10A and 10B are diagrams illustrating a color determination method according to the present embodiment. FIG. 2 is a diagram illustrating a processing flow of the reading system according to the embodiment. 10A and 10B are diagrams illustrating an example of a display of an output result of the reading system according to the present embodiment. 1 is a diagram illustrating an outline of an information processing system that utilizes a reading system according to an embodiment of the present invention.

The present invention is described below, but is not limited to the following content and can be modified as desired without significantly impairing the effects of the present invention. The present invention can also be implemented by combining different embodiments. In the following description, the same components in different embodiments will be designated by the same reference numerals, and duplicate explanations will be omitted.

<Two-dimensional code>
First, a two-dimensional code applied to a reading system according to an embodiment of the present invention will be described. While multiple two-dimensional codes can be applied to this reading system, the following embodiment will be described using a QR Code (registered trademark) as an example of a two-dimensional code. QR Codes are standardized by ISO/IEC 18004 and the like. It should be noted that PDF417, DataMatrix, Maxicode, AztecCode, and the like can also be used as two-dimensional codes.

1A is a schematic diagram of a label display object 100 on which a two-dimensional code 200 (first two-dimensional code) according to this embodiment is arranged. The label display object 100 is composed of the two-dimensional code 200 and a serial ID 101. The two-dimensional code 200 includes three position detection finder patterns 300 (positioning patterns) arranged at the three corners of the two-dimensional code 200, a first reference color 301, a second reference color 302, and a third reference color 303 arranged within the three finder patterns 300, cells 500 constituting a data area, and an alignment pattern 400 that corrects misalignment of the cells 500 caused by distortion. The two-dimensional code 200 further includes an environment detection area 600 arranged to overlap the data area composed of the cells 500. The environment detection area 600 is an area whose color changes in response to environmental changes. The position detection finder patterns 300 are colored with a reference color. In Figure 1A, three finder patterns 300 are each assigned a different reference color (first reference color 301 to third reference color 303). Here, the reference colors are used to determine the amount of color change depending on the shooting environment.

By arranging the first reference color 301, second reference color 302, third reference color 303, and environment detection area 600 all within the two-dimensional code 200, distortion of the two-dimensional code can be reduced when the two-dimensional code is read by the reading system 700 (described below) compared to when they are arranged outside the two-dimensional code, allowing the two-dimensional code itself to be made smaller.

Furthermore, by arranging the first reference color 301, second reference color 302, third reference color 303, and environment detection area 600 all within the two-dimensional code 200, the reading speed can be increased when reading the two-dimensional code using the reading system 700 described below, since the two-dimensional code, reference color, and environment detection area are located closer together (because they are located inside) than when they are arranged outside the two-dimensional code. For example, the reference color can be arranged in the start pattern or stop pattern when using PDF417 instead of a QR code, in the alignment pattern or clock pattern when using DataMatrix, or in the central finder pattern when using Maxicode or AztecCode.

The two-dimensional code 200 can be arranged in a form similar to the label display item 100 and used in the form of a sticker. A sticker-type label is preferred for ease of handling and installation and cost reasons, but it can also be used in any shape, such as a tag or card, depending on the product being managed.

The serial ID 101 attached to the label display item 100 is an ID number linked to the two-dimensional code 200, and can be written anywhere on the label display item 100, making it possible to visually manage the two-dimensional code 200. It is desirable to position the serial ID 101 with a space (quiet zone) of four cells away from the two-dimensional code 200.

The two-dimensional code 200 in Figure 1A is a version 2 (25 x 25 cells) QR code. The cell configuration (version) of the QR code can be changed depending on the information you want to input, but version 2 or higher is preferred because it includes an alignment pattern 400 for distortion correction.

The two-dimensional code 200 in Figure 1A is a QR code with an error correction level of "H." The error correction level of a QR code can be lowered depending on the amount of information and the application, but it is desirable to set the error correction level to "H" by placing an environment detection area 600 within the QR code to prevent reading errors if the QR code is intentionally missing or if it becomes stained, dirty, or damaged during the logistics process.

The environmental detection area 600 placed within the two-dimensional code 200 is not limited to a medium such as ink that changes color in response to environmental changes, but from a manufacturing process perspective, it is preferable that it be in the form of ink that can be printed on the two-dimensional code 200. Examples of environmental detection areas 600 include those that reflect the results of detecting environmental conditions such as temperature, temperature history, humidity, light, gas concentration, and vibration, as well as the pH of a liquid, the concentration of various ions in a liquid, the concentration of various pharmaceuticals, the concentration of various amino acids and proteins, and the presence of viruses and bacteria, as color information.

In this embodiment, the RGB (Red, Green, Blue) model is used as the color model for explanation. In the RGB model, the colors that make up an image are expressed as a combination of values ranging from 0 (the darkest) to 255 (the brightest) for each of the three colors R (red), G (green), and B (blue). Hereinafter, the RGB components will be expressed in square brackets as [R, G, B], for example, red [255, 0, 0], darkest black [0, 0, 0], and brightest white [255, 255, 255]. Note that black or a color equivalent to black is used for the color of untouched colored parts in the two-dimensional code 200 in this embodiment.

The first reference color 301, second reference color 302, and third reference color 303 provided inside the three finder patterns 300 arranged at the three corners of the two-dimensional code 200 are used for color correction of the environment detection area 600 (correction of color information of the environment detection area 600) according to the shooting environment, and are fixed colors (unlike the color of the environment detection area 600, they do not change) colored with ordinary color ink or the like. Examples of the three reference colors are shown below.
The first reference color 301 is painted red [255,0,0].
The second reference color 302 is painted green [0,255,0].
The third reference color 303 is painted blue [0,0,255].

When printing a two-dimensional code 200 in color, if full color is expressed using the four colors CMYK (cyan, magenta, yellow, black), there are areas that can be reproduced in RGB but cannot be reproduced in CMYK. Therefore, in the areas that can be reproduced in CMYK, there is no particular problem if red is used as the first reference color 301, green as the second reference color 302, and blue as the third reference color 303. Furthermore, while the possible combinations of reference colors are not limited to red, green, and blue, it is preferable to select three colors that are located at different positions in the RGB color space. Specifically, it is preferable that each of the three finder patterns be colored with a reference color that has a different vector direction connecting it to white (255, 255, 255) in the RGB space.

Figure 1B is an enlarged view of finder pattern 300 within two-dimensional code 200. Finder pattern 300 is composed of first finder pattern region 310, second finder pattern region 320, and third finder pattern region 330. This finder pattern allows the position of the two-dimensional code to be recognized, enabling high-speed reading. In this embodiment, a reference color is assigned within finder pattern 300, but the finder pattern must retain the function of recognizing the position of the two-dimensional code. From any position in the A, B, or C directions of finder pattern 300, the ratio of white (light color) cells to black (dark color) cells must be arranged in a 1:1:3:1:1 ratio. Color can be assigned to any part within the above rules. There are no particular restrictions on the reference colors placed in the first finder pattern area 310, the second finder pattern area 320, and the third finder pattern area 330, as long as they are colors that can be read by a two-dimensional code reader; however, the colors used in the first finder pattern area 310 and the third finder pattern area 330 are preferably colors that are recognized as being similar to dark colors or low-brightness black, and the colors used in the second finder pattern area 320 are preferably colors that are recognized as being similar to light colors or high-brightness white cells.

Figure 1C is an enlarged view of the alignment pattern 400 in the two-dimensional code 200. The alignment pattern has the function of correcting misalignment of each cell caused by distortion. The alignment pattern is composed of a first alignment pattern area 410, a second alignment pattern area 420, and a third alignment pattern area 430. Each of these alignment pattern areas can also have the first reference color 301, the second reference color 302, and the third reference color 303, as well as an environment detection area 600. However, it is necessary to maintain the function of correcting misalignment of cells caused by distortion. There are no particular restrictions on the reference colors placed in the first finder pattern region 310, second finder pattern region 320, and third finder pattern region 330, as long as they are colors that can be read by a two-dimensional code reader; however, the colors used in the first alignment pattern region 410 and third alignment pattern region 430 are preferably colors that are recognized as dark or low-brightness black, and the colors used in the second alignment pattern region 420 are preferably colors that are recognized as light or high-brightness white cells.

Specific embodiments of two-dimensional codes that can be used in the reading system of an embodiment of the present invention are shown below in Figures 1D to 1L. Figures 1D to 1L are schematic diagrams of two-dimensional codes (second to tenth two-dimensional codes) that are different from the two-dimensional code (first two-dimensional code) used in the display object of Figure 1A.

The two-dimensional code 201 (second two-dimensional code) shown in Figure 1D is a QR code in which a first reference color 301 and a second reference color 302 are placed in the first finder pattern area 310 (see Figure 1B) of each of two of the three finder patterns 300 arranged at the three corners of the QR code. The two finder patterns to be colored with reference colors can be selected arbitrarily, and the reference color to be colored in the first finder pattern area 310 can be set arbitrarily.

The two-dimensional code 202 (third two-dimensional code) shown in Figure 1E is a QR code in which a first reference color 301 is placed in the first finder pattern area 310 (see Figure 1B) of one of the three finder patterns 300 located at the three corners of the QR code. The one finder pattern to be assigned the reference color can be selected arbitrarily, and the reference color to be assigned to the first finder pattern area 310 can be set arbitrarily. As described above, the reference color only needs to be assigned to one or more of the three finder patterns.

The two-dimensional code 203 (fourth two-dimensional code) shown in Figure 1F is a QR code in which a first reference color 301 is placed in the first finder pattern areas 310 (see Figure 1B) of all three finder patterns 300 located at the three corners of the QR code. The reference color to be placed in the first finder pattern area 310 can be set arbitrarily. As mentioned above, the reference colors of the finder patterns may be the same color. As mentioned above, it is desirable to select three colors that are located at different positions in the RGB color space.

The two-dimensional code 204 (fifth two-dimensional code) shown in Figure 1G is a QR code in which a first reference color 301, a second reference color 302, and a third reference color 303 are arranged in the third finder pattern area 330 (see Figure 1B) of three finder patterns 300 arranged at the three corners of the QR code. The reference colors to be arranged in the third finder pattern area 330 can be set arbitrarily.

The two-dimensional code 205 (sixth two-dimensional code) shown in Figure 1H is a QR code in which a first reference color 301, a second reference color 302, and a third reference color 303 are arranged in the first finder pattern area 310 (see Figure 1B) and the third finder pattern area 330 (see Figure 1B) of three finder patterns 300 arranged at the three corners of the QR code. The reference colors to be arranged in the first finder pattern area 310 and the third finder pattern area 330 can be set arbitrarily.

The two-dimensional code 206 (seventh two-dimensional code) shown in Figure 1I is a QR code in which a first reference color 301, a fourth reference color 304, and a third reference color 303 are arranged in the first finder pattern region 310 (see Figure 1B), the second finder pattern region 320 (see Figure 1B), and the third finder pattern region 330 (see Figure 1B) of three finder patterns 300 arranged at the three corners of the QR code. The reference colors and colors to be assigned to the first finder pattern region 310, the second finder pattern region 320, and the third finder pattern region 330 can be set arbitrarily, but it is desirable that the fourth reference color 304 assigned to the second finder pattern region 320 be a light color such as gray [200, 200, 200] that is close to white.

The two-dimensional code 207 (eighth two-dimensional code) shown in Figure 1J is a QR code in which a fourth reference color 304 is placed in the second alignment pattern region 420 (see Figure 1C) of the QR code alignment pattern 400. This color arrangement allows the fourth reference color 304 to be placed so as to surround four environment detection regions 600 that are located at different positions in the RGB color space, thereby improving the color accuracy of the environment detection region. The reference colors to be placed in the first finder pattern region 310 (see Figure 1B), second finder pattern region 320 (see Figure 1B), third finder pattern region 330 (see Figure 1B), and second alignment pattern region 420 (see Figure 1C) can be set arbitrarily.

The two-dimensional code 208 (ninth two-dimensional code) shown in Figure 1K is a QR code in which an environment detection area 600 is located in a different position from the QR code. The environment detection area 600 can be located anywhere within the two-dimensional code, as long as it does not overlap with the finder pattern 300 or the alignment pattern 400. The environment detection area 600 can also be expanded up to 30% of the area occupied by the cell 500, and can be located in any size and position as long as the two-dimensional code can be read using a commercially available two-dimensional code reader or smartphone camera.

The two-dimensional code 209 (tenth two-dimensional code) shown in Figure 1L is a two-dimensional code in which the environment detection region 600 of the two-dimensional code 200 (first two-dimensional code) is colored in the first alignment pattern region 410 of the alignment pattern 400. By utilizing the alignment pattern 400 as the environment detection region 600, it is possible to prevent loss of cells 500 in the two-dimensional code, making it easier to obtain information from the two-dimensional code. Using a similar concept, an environment detection region 600 can also be provided in the finder pattern 300. When providing an environment detection region 600 in either the alignment pattern 400 or the finder pattern 300, it is necessary to provide an environment detection region that can maintain the functionality of the alignment pattern 400 or the finder pattern 300 before and after the color of the environment detection region 600 changes.

<Reading system>
Next, a reading system according to an embodiment of the present invention will be described.
2 is a configuration diagram of a two-dimensional code reading system 700. The reading system 700 is used to read, for example, the two-dimensional codes shown in FIGS. 1A and 1C to 1L. The reading system 700 includes an image capture device 710 that captures an image of the two-dimensional code, an input device 720, an output device 730, a data processing device 740, and a storage device 760.

The image capture device 710 is an imaging device such as a camera that captures images of label items. If necessary, it is also possible to capture images of memorized products, product-related information, the surrounding environment, and other items in addition to label items. Once an image of the two-dimensional code is captured, it is stored in the image data storage unit 761 in the storage device 760.

The input device 720 is the part that accepts instructions from the operator and is composed of buttons, a touch panel, etc.

The output device 730 is a device that outputs instruction information, scanned images, scanning results, etc. to the operator, and is composed of a display and communication device. This configuration is standard, but any or all of the image acquisition device 710, input device 720, and output device 730 may be configured to be connected externally to the scanning system 700.

(Storage device)
The storage device 760 is a part that stores various types of data, and is composed of the following storage units.

The image data storage unit 761 stores the image of the two-dimensional code input from the image acquisition device 710.

The two-dimensional code position data storage unit 762 stores data representing the reference position of the code recognized by the two-dimensional code position recognition unit 742 (described below) from the image stored in the image data storage unit 761.

The two-dimensional code data storage unit 763 stores data of a character string represented by a code recognized by the two-dimensional code recognition unit 743 (described below) from an image stored in the image data storage unit 761.

The reference color position data storage unit 764 is a part that stores in advance the position information of the reference color placed within the two-dimensional code.

The environment detection area position data storage unit 765 is a part that stores in advance the position information of the environment detection area installed within the two-dimensional code.

The reference color measured color data storage unit 766 stores measured color information (RGB values, etc.) of the reference color extracted from the reference color area identified by the reference color position data storage unit 764.

The environment detection area measured color data storage unit 767 stores measured color information (RGB values, etc.) of the environment detection area extracted from the area of the environment detection area identified by the environment detection area position data storage unit 765.

The reference color base data memory unit 768 is a unit that stores color information (RGB values, etc.) of the reference color that serves as the basis for correcting the measured color information (RGB values, etc.) of the reference color.

The corrected color data storage unit 769 for the environment detection area stores color information (RGB values, etc.) that has been corrected from the actual color information (RGB values, etc.) of the environment detection area stored in the image data storage unit 761.

The environment detection area judgment color data storage unit 770 stores, for example, threshold color information (RGB values, etc.) that serves as the judgment standard for determining whether or not a temperature deviation has occurred, and the relationship between the amount of environmental change and the color information of the environment detection area, based on the correction color of the environment detection area stored in the correction color data storage unit 769 for the environment detection area. The relationship between the amount of environmental change and the color information of the environment detection area indicates, for example, the ambient temperature corresponding to a certain color in the environment detection area.

The judgment result storage unit 771 stores the results, such as "NG"/"OK" or "■"/"■", determined by the judgment unit 749 (described below) based on the magnitude of the RGB values stored in the correction color data storage unit 769 for the environment detection area relative to the RGB values stored in the judgment color data storage unit 770 for the environment detection area.

The read data storage unit 772 is a unit that stores data other than that processed by the data processing device 740, which will be described later. Generally, when understanding changes in the product's environment, it is desirable to have all the information about what happened, when, and where. Therefore, in addition to the product-related information stored in the 2D code data storage unit 763 and data such as temperature deviations from the environmental detection area stored in the determination result storage unit 771, it is preferable to also store the date and time the read process was performed, location information, the number of the reading device (to identify the worker), and weather information linked to web information. Note that as long as the same reading device is used, the reading device number will not change, so it may be added when outputting data, which will be described later.

(Data Processing Device)
The data processing device 740 processes data input from the image acquisition device 710 and the input device 720 and data stored in the memory device 760, and outputs the results to the output device 730 or stores them in the memory device 760, and is composed of the following processing units.

The input control unit 741 separates data input from the image acquisition device 710 or input device 720 into commands, data, etc., and transfers them to the storage device 760 and various parts of the data processing device 740. In particular, the main data transferred to the image data storage unit 761 is image data of a two-dimensional code including a reference color and an environment detection area.

The two-dimensional code position recognition unit 742 recognizes the position of the code from the image data stored in the image data storage unit 761, and stores the recognition results in the two-dimensional code position data storage unit 762. Image data is typically made up of hundreds or thousands of dots both vertically and horizontally, and the image itself does not contain data indicating which parts are codes. Therefore, the color data of each dot is analyzed to recognize which parts of the image data correspond to the reference position of the code. The display format and number of reference positions vary depending on the code standard, but are not limited to these standards. Furthermore, even if there are multiple two-dimensional codes in the image stored in the image data storage unit 761, the positions of the multiple two-dimensional codes can be recognized.

The two-dimensional code recognition unit 743 uses the position data stored in the two-dimensional code position data storage unit 762 to recognize character string data represented by a code from the image stored in the image data storage unit 761, and stores the data in the two-dimensional code data storage unit 763.

The reference color position determination unit 744 is a unit that uses character string data to identify the type of two-dimensional code from the position data stored in the two-dimensional code position data storage unit 762 and the information stored in the two-dimensional code data storage unit 763, detects the coordinates (X, Y) of the four corners of the two-dimensional code from the captured image of the two-dimensional code, and determines the positions of the reference colors (first reference color 301 to fourth reference color 304) based on the coordinates of the detected four corners and the position information previously stored in the reference color position data storage unit 764. Here, identifying the type of two-dimensional code means, for example, identifying which type of two-dimensional code it is, as shown in Figure 1A or Figures 1D to 1L.

The environment detection area position determination unit 745 uses character string data to identify the type of two-dimensional code from the position data stored in the two-dimensional code position data storage unit 762 and the information stored in the two-dimensional code data storage unit 763, detects the coordinates (X, Y) of the four corners of the two-dimensional code from the captured image of the two-dimensional code, and determines the position of the environment detection area 600 based on the coordinates of the detected four corners and the position information previously stored in the environment detection area position data storage unit 765.

The reference color measured color determination unit 746 is a part that determines the RGB values of the reference colors in the acquired image from the image stored in the image data storage unit 761 and the position information of the reference colors (first reference color 301 to fourth reference color 304) determined by the reference color position determination unit 744.

The environment detection area measured color determination unit 747 is a part that determines the RGB values of the environment detection area 600 in the acquired image from the image stored in the image data storage unit 761 and the position information of the environment detection area 600 determined by the environment detection area position determination unit 745.

The corrected color determination unit 748 of the environment detection area is a unit that determines RGB values by correcting the color tones that have changed due to the lighting environment to the original colors from the measured RGB values in the acquired image determined by the measured color determination unit 747 of the environment detection area.

The method for determining the correction color of the environment detection area is described below.
FIG. 3 is a diagram illustrating a color correction method according to this embodiment. FIG. 3 illustrates a method for correcting the measured RGB values of the environment detection area. The environment detection area correction color determination unit 748 calculates the difference between the RGB values stored in the reference color base data storage unit 768 and the RGB values of the reference color in the acquired image stored in the reference color measured color data storage unit 766, and applies the resulting difference in RGB values to the RGB values of the environment detection area in the acquired image stored in the environment detection area measured color data storage unit 767, thereby correcting the measured colors of the environment detection area in the acquired image. The corrected RGB values of the environment detection area are stored in the environment detection area correction color data storage unit 769.

The specific color correction method is shown below.
Here, a method for correcting the color of the environment detection area 600 is shown, using three reference colors: a first reference color 301, a second reference color 302, and a third reference color 303, which are provided inside three finder patterns 300 in the two-dimensional code 200 shown in Figure 1A. There is no limitation on the number of reference colors, and the following formula can be converted depending on the number of reference colors.

Regarding the reference color, the relationship between the original color and the color in the scanned image is expressed as equation (1).
Here, _Rd is the original color of the reference color, that is, the color defined as the reference color and registered in advance in the reference color base data storage unit 768. _Rs is the color of the reference color in the image acquired by the image acquisition device 710. And C is the color change.

The relationship between the reference colors can be written in matrix form as shown in equation (2): where i in _Rdi and _Rsi indicates the number of each of the three reference colors provided.

Additionally, the relationship of formula (3) holds in the environment detection region.
Here, _Ts is the color of the environment detection area in the acquired image stored in the environment detection area measured color data storage unit 767, and _Tc is the value resulting from color correction of this color.

The relationship between the environment detection regions can be expressed in matrix form as shown in equation (4).

Therefore, if C is found from the relationship of the reference colors, then by finding the product of C and the color _Ts of the environment detection area in the acquired image, _Tc , which is the original color of the environment detection area, can be found.

To find C, equation (5) is used.

Returning to Figure 2, the determination unit 749 compares the RGB values stored in the correction color data storage unit 769 of the environment detection area with the RGB values stored in the judgment color data storage unit 770 of the environment detection area, and determines, for example, whether or not there is a temperature deviation based on the magnitude of the vector values of the RGB values in the RGB space of Figures 4A and 4B, which will be described later.

Fig. 4A is a diagram illustrating a color determination method according to this embodiment. Fig. 4A illustrates a first color determination method for determining the color of the environment detection area 600. Equation (6) shows a vector from the origin of white (255, 255, 255) to the measured color (r _m , g _m , b _m ). The measured color here is the color corrected by the reference color using the above method.

Depending on the length of this l, it is determined whether the color exceeds a certain set standard.

The first color determination method for determining the color of the environment detection area 600 shown in Figure 4A can return a color determination by calculating the simple formula (6), but it does not take into account the vector direction in the RGB color space, and will determine that the colors are the same even if the hue is different if the vector length is the same.

Therefore, when the color tones of the environment detection area 600 are different but the values of equation (6) are close, the second determination method shown in Figure 4B can be applied.

Equations (7) and (8) show the vectors of the final color ( _Rf , _Gf , _Bf ) and the measured color ( _rm , _gm , _bm ) with the starting color ( _Ri , _Gi , _Bi ) as the origin, where the measured color is the color corrected by the reference color using the above method.
Final color (R, G, B)=( _Rf - _Ri , _Gf - _Gi , _Bf - _Bi ) (7)
Measured color (r, g, b) = (r _m - R _i , g _m - G _i , b _m - B _i ) ... Equation (8)

From equations (7) and (8), the lengths of the vectors from the starting color to the final color and the measured color (absolute values of R and r, respectively) are expressed by equations (9) and (10).

The length from the measured color (r _m , g _m , b _m ) to the shifted color (r _M , g _M , b _M ) obtained by drawing a perpendicular line on the vector of equation (7) is expressed by equation (11).

The inner product of the vectors in equations (9) and (10) is expressed as equation (12).

The ratio of the length of the vector from the starting color to the correction color (formula (10) above) to the length of the vector from the starting color to the final color (formula (9) above) is calculated by substituting formulas (11) and (12) into formulas (9) and (10) to obtain formula (13).

Returning to Figure 2, the output control unit 750 controls the output of information processed by the data processing device 740 to the output device 730. Specifically, this is the part that outputs to the output device 730 product information linked to the character string stored in the two-dimensional code data storage unit 763, information such as the presence or absence of temperature deviation determined from the color information of the environment detection area stored in the determination result storage unit 771, or information stored in the read data storage unit 772. When the output destination is a screen, etc., it is preferable to output the result each time a reading operation is performed. In this case, it is also preferable to output the determination result stored in the determination result storage unit 771 for the environment detection area. When the output destination is a communication destination, etc., the output process may be performed each time a reading operation is performed, or may be performed by aggregating data from several readings or at predetermined intervals.

<Processing flow of the reading system>
5 is a diagram illustrating a process flow T800 of the reading system 700 according to this embodiment. The process flow T800 will be described below in conjunction with FIG.

First, based on a command from the input control unit 741, image data of a two-dimensional code, including a reference color and an environment detection area, is input from the image acquisition device 710. The input data is stored in the image data storage unit 761. The two-dimensional code position recognition unit 742 recognizes the reference position of the two-dimensional code from the image recorded in the image data storage unit 761, and the recognized reference position is recorded in the two-dimensional code position data storage unit 762. The two-dimensional code recognition unit 743 uses the image recorded in the image data storage unit 761 and the position data recorded in the two-dimensional code position data storage unit 762 to identify the data area and decode the code represented by the data area. The two-dimensional code recognition unit 743 uses the character string data obtained by decoding to identify the type of two-dimensional code. (Procedure T801)

In step T802, based on the type of identified two-dimensional code, the position information of the reference color and environment detection area of the two-dimensional code corresponding to the identified type of two-dimensional code is retrieved from the storage device 760, thereby identifying the position information and color information of the reference color and environment detection area located within the two-dimensional code. The identification results are recorded in the two-dimensional code data storage unit 763.

Using the position data stored in the 2D code position data storage unit 762 and the information stored in the 2D code data storage unit 763, the coordinates (X, Y) of the four corners of the 2D code are detected from the captured image of the 2D code. The reference color position determination unit 744 and the environment detection area position determination unit 745 use the coordinates of the detected four corners and the position information of the reference color and environment detection area previously stored in the reference color position data storage unit 764 and the environment detection area position data storage unit 765 to determine the positions of the colors to be used in the analysis for the reference color and the environment detection area, respectively. (Procedure T802)

The reference color measurement color determiner 746 and the environment detection area measurement color determiner 747 determine color information for each reference color area and environment detection area from the acquired image based on the position information determined by the reference color position determiner 744 and the environment detection area position determiner 745. Here, color information refers to, for example, RGB values. The determined color information is transferred to the reference color measurement color data memory 766 and the environment detection area measurement color data memory 767, which store the color information for the reference color area and the environment detection area, respectively. If strong lighting causes "overexposure" (where the reference color or environment detection area is obscured by reflected light), if there is an "extreme illuminance difference" between multiple reference colors or between the reference color and the environment detection area, if the 2D code image is too small or too large, or if the shooting angle during image capture is acute and the 2D code image is excessively distorted, return to step T801 and perform 2D code recognition and image scanning again. (Procedure T803)

If the measured color data for the reference color and environment detection area can be acquired without any problems in step T803, color correction for the environment detection area (correction of the color information for the environment detection area) is then performed. Details are as explained above. Using the color information stored in the reference color base data storage unit 768, the reference color measured color data storage unit 766, and the environment detection area measured color data storage unit 767, the environment detection area correction color determination unit 748 determines the correction color for the environment detection area and stores it in the environment detection area correction color data storage unit 769. (Step T804)

Then, as described above, the determination unit 749 compares the color information stored in the correction color data storage unit 769 for the environment detection area with the color information stored in the judgment color data storage unit 770 for the environment detection area to determine whether the amount of environmental change has exceeded a preset threshold. In other words, the data processing device 740 uses the color information of the corrected environment detection area to determine whether the surrounding environment of the 2D code has exceeded a preset threshold. The determination unit 749 may calculate the surrounding environment information for the 2D code from the color information of the corrected environment detection area and the relationship between the amount of environmental change and the color information of the environment detection area stored in the judgment color data storage unit 770 for the environment detection area. (Step T805)

The results of the determination made by the determination unit 749 are stored in the determination result storage unit 771. At the same time, the read data storage unit 772 stores the date and time of the reading process, location information, the reading device number (for identifying the worker), weather information linked to web information, and other information. (Step T806)

Finally, the product information associated with the character string stored in the 2D code data storage unit 763, information such as the presence or absence of temperature deviation determined from the color information of the environmental detection area stored in the determination result storage unit 771, or information stored in the read data storage unit 772 are output to the output device 730. If a smartphone is used as hardware for the reading system, the output device 730 outputs the reading results to the smartphone screen or other device. (Step T807)

Figure 6 is a diagram showing an example of the display of the output results of the reading system 700 according to this embodiment. Referring to Figure 6, an example of the output display of the reading results on the display 901 of the smartphone 900 is shown. When multiple label objects are lined up on the display 901, it is possible to simultaneously acquire images of multiple items and display the reading results. Specifically, considering on-site workability and processing speed, it is efficient and preferable to recognize up to eight two-dimensional codes, process them in parallel using four threads in the order they are recognized, and then, once processing is complete, process the remaining two-dimensional codes in parallel. The number of parallel processing threads is not particularly limited, as the optimal number will vary depending on the on-site work and the processing speed of the smartphone 900.

The results determined in step T805 are displayed as "■" 902a, which indicates a temperature deviation in AR (Augmented Reality) and is NG, or "■" 902b, which indicates a temperature deviation in AR and is OK, making it easier for the worker to recognize the management status. After all two-dimensional codes have been read, by touching the end processing button 903, all acquired data output in step 802 can be viewed on the smartphone screen or sent to a specified address.

In this embodiment, processing flow T800 is preferably executed by a general-purpose smartphone or similar device equipped with a camera, screen, and communication device, but is not limited to this form.

<Information Processing System>
Fig. 7 is a diagram illustrating an outline of an information processing system 1000 that utilizes a reading system 700 according to this embodiment. Fig. 7 shows a block diagram of the information processing system 1000 that includes an information processing device 1100. Although not shown, the information processing device 1100 is configured to include, for example, a central processing unit (CPU), random access memory (RAM), read only memory (ROM), a hard disk drive (HDD), an interface (I/F), and the like. The information processing device 1100 is realized by the CPU executing a predetermined control program stored in the ROM.

The information processing device 1100 is composed of an input unit 1110, a memory unit 1120, an information processing unit 1130, and an output unit 1140. The input unit 1110 is a unit that inputs information output from the smartphone 900. The memory unit 1120 is a unit that accumulates and stores information output from the smartphone 900.

The information processing unit 1130 organizes and analyzes the information stored in the memory unit 1120. For example, by statistically analyzing information from the smartphone 900 that includes the reading system 700, it is possible to determine when, which worker, and in what location a temperature deviation occurred, leading to operational improvements.

The output unit 1140 is a part that transmits the information processed by the information processing unit 1130 to the location where it is needed. If the information processing device 1100 is a device that has a display, it can output the processing results to that display.

The following is an example of applying the information processing system 1000 to distribution from production to consumers. The information processing system 1000 enables consumers who receive a product to understand the physical quantities of a product, such as wine, during the distribution process. For example, if the product is wine, the two-dimensional code 200 is displayed on the label attached to the wine, the box containing the wine, etc. Figure 7 uses the two-dimensional code 200 as an example, but the other two-dimensional codes 201-209 can be used in a similar manner.

The information processing system 1000 includes a smartphone 900 having a reading system 700 for the two-dimensional code 200, and an information processing device 1100. The smartphone 900 (input/output device), which is typically a smartphone, is a terminal with an imaging function, such as an information communication terminal. The smartphone 900 (input/output device) is used to upload information acquired and analyzed by the reading system 700 for the two-dimensional code 200 to the information processing device 1100 during product distribution. Figure 7 shows an example of a smartphone 900 (input/output device) that is capable of both input and output on a single terminal, but dedicated input and output devices can also be used.

When collecting products, the person in charge of transporting the products (e.g., a driver) uses a smartphone 900 (input/output device) to capture an image of the two-dimensional code 200 displayed on the product. This uploads input information including an image of the two-dimensional code 200 to the information processing device 1100. The two-dimensional code 200 and reading system 700 determine whether there is a temperature deviation at the time of reading. At this time, in addition to product identification information (item identification information), the image capture time, image capture location, etc. are also uploaded as input information. Examples of item identification information include GTIN (Global Trade Item Number), EAN (European Article Number) code, and U.P.C. (Universal Product Code). It is preferable that this item identification information be linked to information about the item, such as the manufacturer, seller, production date, and expiration date.

In the same way, each person in charge at the export warehouse, customs warehouse, and import warehouse uses a smartphone 900 (input/output device) to capture an image of the two-dimensional code 200 displayed on the product. This uploads the input information to the information processing device 1100. The information processing device 1100 then determines whether or not there was a temperature deviation at the time of reading based on the uploaded input information.

On the other hand, consumers can access the URL (web) stored in the two-dimensional code 200 by reading the product's two-dimensional code 200 using a smartphone 900 (input/output device). Reading can be performed, for example, using a dedicated application running on the smartphone 900 (input/output device).

Note that the smartphone 900 on the manager's side, such as the person in charge of product transportation, includes the reading system 700 of this embodiment, but the smartphone 900 on the consumer's side may not include the reading system 700 of this embodiment. While Figure 7 shows the smartphone 900 as a smartphone or tablet terminal, this is not limited to this, and the smartphone 900 may also be a dedicated reading reader as an input device and a personal computer as an output device.

The two-dimensional code reading system 700 of this embodiment is a two-dimensional code reading system having an ink in the data area that changes color in response to environmental changes. The two-dimensional code has a positioning pattern colored with a reference color "used to determine the amount of color change in response to the shooting environment." The system is equipped with an image acquisition device 710 that acquires an image of the two-dimensional code, a data processing device 740, and a storage device 760 that stores color information of the reference color and position information of the reference color and ink. The data processing device 740 detects the coordinates (X, Y) of the four corners of the two-dimensional code from the image of the two-dimensional code, identifies the position of the area where color information is to be detected based on the detected coordinates of the four corners, and detects color information of the area. The area includes a reference color area and an ink area. The reference color area and ink area are determined (identified) from the position information of the reference color stored in the storage device 760 and the color information and position of the area, and the color information of the ink area can be corrected using the relationship between the color information of the reference color stored in the storage device 760 and the color information of the detected reference color area. Note that the ink area whose color changes in response to environmental changes corresponds to the environment detection area 600. This improves the accuracy and reading speed of obtaining color information about ink whose color changes in response to environmental changes.

100 Label display 101 Serial ID
200 Two-dimensional code (first two-dimensional code)
201 Two-dimensional code (second two-dimensional code)
202 Two-dimensional code (third two-dimensional code)
203 Two-dimensional code (fourth two-dimensional code)
204 Two-dimensional code (fifth two-dimensional code)
205 Two-dimensional code (6th two-dimensional code)
206 Two-dimensional code (7th two-dimensional code)
207 Two-dimensional code (8th two-dimensional code)
208 Two-dimensional code (9th two-dimensional code)
209 Two-dimensional code (10th two-dimensional code)
300 Finder pattern (positioning pattern)
301 First reference color 302 Second reference color 303 Third reference color 304 Fourth reference color 310 First finder pattern area 320 Second finder pattern area 330 Third finder pattern area 400 Alignment pattern 410 First alignment pattern area 420 Second alignment pattern area 430 Third alignment pattern area 500 Cell (data area)
600 Environmental detection area 700 Reading system (two-dimensional code reading system)
710 Image acquisition device 720 Input device 730 Output device 740 Data processing device 741 Input control unit 742 Two-dimensional code position recognition unit 743 Two-dimensional code recognition unit 744 Reference color position determination unit 745 Environment detection area position determination unit 746 Reference color measured color determination unit 747 Environment detection area measured color determination unit 748 Environment detection area correction color determination unit 749 Determination unit 750 Output control unit 760 Storage device 761 Image data storage unit 762 Two-dimensional code position data storage unit 763 Two-dimensional code data storage unit 764 Reference color position data storage unit 765 Environment detection area position data storage unit 766 Reference color measured color data storage unit 767 Environment detection area measured color data storage unit 768 Reference color base data storage unit 769 Environment detection area correction color data storage unit 770 Environment detection area judgment color data storage unit 771 Judgment result storage unit 772 Read data storage unit 900 Smartphone (mobile terminal)
901 Display 902a AR display "NG"
902b AR display “OK”
903 Processing end button 1000 Information processing system 1100 Information processing device 1110 Input unit 1120 Storage unit 1130 Information processing unit 1140 Output unit T800 Processing flow of reading system T801 Procedure "Code recognition/image reading"
T802 Procedure "Obtaining reference color and ink color"
T803 Procedure "Burst highlights, extreme illumination differences, image size, image angle"
T804 Procedure "Color Correction"
T805 Procedure "Color Analysis"
T806 Procedure "Data Storage"
T807 Procedure "Data output/transmission"

Claims (10)

A two-dimensional code reading system having an environment detection area in a data area that changes color in response to environmental changes,
The method includes an image capture device that captures an image of the two-dimensional code, a data processing device, and a storage device that stores color information of a reference color and position information of the reference color and an environment detection area,
the two-dimensional code has a positioning pattern colored with a reference color used to determine the amount of color change depending on the shooting environment,
The data processing device includes:
Recognizing a reference position of the two-dimensional code from an image of the two-dimensional code;
determining an area colored with the reference color and the environment detection area in an image of the two-dimensional code acquired from the detected reference position and the position information of the reference color and the environment detection area stored in the storage device, and detecting color information of the reference color and the environment detection area;
a two-dimensional code reading system, characterized in that the color information of the environment detection area is corrected using the relationship between the color information of the reference color stored in the storage device and the color information of the detected reference color area. The two-dimensional code reading system according to claim 1,
The data processing device identifies the type of the two-dimensional code based on the information stored in the data area of the two-dimensional code, and retrieves the reference color of the two-dimensional code and the position information of the environment detection area corresponding to the identified type of two-dimensional code from the storage device. The two-dimensional code reading system according to claim 1,
the storage device stores a relationship between an amount of change in the environment and color information of the environment detection area;
The data processing device calculates ambient environment information of the two-dimensional code from the corrected color information of the environment detection area and the relationship between the amount of environmental change and the color information of the environment detection area stored in the storage device. The two-dimensional code reading system according to claim 1,
The data processing device uses the corrected color information of the environment detection area to determine whether the surrounding environment of the two-dimensional code exceeds a preset threshold value. The two-dimensional code reading system according to claim 1,
the image acquisition device acquires an image including a plurality of the two-dimensional codes;
The two-dimensional code reading system is characterized in that the data processing device performs processing for recognizing a plurality of the two-dimensional codes in parallel. The two-dimensional code reading system according to claim 1,
The two-dimensional code reading system, wherein the environment detection area is formed of ink that changes color in response to environmental changes. The two-dimensional code reading system according to claim 1,
the two-dimensional code is a QR code and has three of the positioning patterns;
A two-dimensional code reading system characterized in that each of the three positioning patterns is colored with a reference color having a different vector direction connected to white (255, 255, 255) in the RGB space. A two-dimensional code having a data area made up of a plurality of cells and a positioning pattern,
an environment detection area that changes color in response to environmental changes, the environment detection area being superimposed on the data area;
the positioning pattern has a portion colored with a reference color used to calculate a color change amount according to a shooting environment,
the two-dimensional code has three of the positioning patterns,
Each of the three positioning patterns is colored with a reference color that has a different direction of the vector connecting with white (255, 255, 255) in the RGB space.
A two-dimensional code characterized by: The two-dimensional code according to claim 8 ,
The two-dimensional code is characterized in that the environment detection area is formed of ink that changes color in response to environmental changes. The two-dimensional code according to claim 8 ,
the reference colors include a first reference color and a second reference color that are different from each other;
A two-dimensional code, wherein the positioning pattern has a portion marked with a first reference color and a portion marked with a second reference color.