WO2019177343A1 - Réfrigérateur, et système et son procédé de commande - Google Patents

Réfrigérateur, et système et son procédé de commande Download PDF

Info

Publication number
WO2019177343A1
WO2019177343A1 PCT/KR2019/002851 KR2019002851W WO2019177343A1 WO 2019177343 A1 WO2019177343 A1 WO 2019177343A1 KR 2019002851 W KR2019002851 W KR 2019002851W WO 2019177343 A1 WO2019177343 A1 WO 2019177343A1
Authority
WO
WIPO (PCT)
Prior art keywords
food
image
user
refrigerator
touch
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2019/002851
Other languages
English (en)
Korean (ko)
Inventor
강병익
한성주
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from KR1020190023124A external-priority patent/KR102217024B1/ko
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Priority to US16/979,586 priority Critical patent/US11425338B2/en
Priority to EP19766651.4A priority patent/EP3745304B1/fr
Priority to CN201980025477.6A priority patent/CN111971684A/zh
Publication of WO2019177343A1 publication Critical patent/WO2019177343A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
    • F25—REFRIGERATION OR COOLING; COMBINED HEATING AND REFRIGERATION SYSTEMS; HEAT PUMP SYSTEMS; MANUFACTURE OR STORAGE OF ICE; LIQUEFACTION SOLIDIFICATION OF GASES
    • F25D—REFRIGERATORS; COLD ROOMS; ICE-BOXES; COOLING OR FREEZING APPARATUS NOT OTHERWISE PROVIDED FOR
    • F25D29/00—Arrangement or mounting of control or safety devices
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/24—Aligning, centring, orientation detection or correction of the image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features

Definitions

  • the disclosed invention relates to a refrigerator, a control method and a system thereof, and more particularly, to a refrigerator capable of identifying food stored in the refrigerator, a control method and a system thereof.
  • the refrigerator is a device for long-term storage of the stored items without decay, such as food, drinks, etc., and is usually provided with a refrigerating chamber for cold storage of the storage and a freezing chamber for freezing the storage.
  • the refrigerator repeats the cooling cycle, including compression-condensation-expansion-evaporation of the refrigerant, to maintain the temperature in the storage compartment at the set target temperature.
  • the refrigerator supplies the air cooled by the evaporator provided corresponding to each storage compartment based on the target temperature of each storage compartment (the refrigerator compartment and / or the freezing compartment) into each storage compartment so that the temperature of the storage compartment is maintained at the target temperature.
  • a refrigerator may include a display for displaying a temperature of a storage compartment and an operation mode of the refrigerator.
  • This display provides the user with a graphical user interface, and allows the user to easily obtain information related to the refrigerator and / or food related information using the graphical user interface and the touch panel, and the user intuitively controls commands. Allow input.
  • the refrigerator may display the information of the food stored in the storage room on the display in response to a user input input through the display, and manage the information of each food.
  • the information of the food stored in the storage room was generally input directly by the user. Therefore, whenever the food is stored in the refrigerator or the food is taken out of the refrigerator, the user is inconvenient to change the information of the food stored in the storage compartment.
  • One aspect of the disclosed invention is to provide a refrigerator, a control method and a system thereof, which can automatically identify food stored in the refrigerator.
  • One aspect of the disclosed invention is to provide a refrigerator, a control method and a system thereof capable of identifying food stored in the refrigerator using machine learning.
  • One aspect of the disclosed invention is to provide a refrigerator, a control method and a system thereof that can provide various services to a user based on food identified using machine learning.
  • Refrigerator according to one aspect of the disclosed disease storage room; A camera provided in the storage compartment; A touch sensitive display configured to receive a user's touch input; A communication unit communicating with an external device; The camera is controlled to photograph the inside of the storage room, the image inside the storage room is displayed on the touch-sensitive display, the communication unit is controlled to transmit the image to the external device, and in response to the user's touch input. And a controller configured to receive identification information from the image from the external device through a communication unit and to display the identification information on the touch-sensitive display.
  • the identification information may include identification information of the food identified from the image and location information in the image of the identified food.
  • the controller may determine a food corresponding to the user's touch input based on the location information of the user's touch input and the location information in the image of the identified food, and the food corresponding to the user's touch input. Information may be displayed on the touch-sensitive display.
  • the location information in the image of the identified food may include the location of a plurality of sample images arbitrarily selected from the image.
  • the identification information of the food identified from the image may include identification information of the food identified from the plurality of sample images.
  • the controller may select a sample image having a minimum distance from the user's touch input among the plurality of sample images.
  • the controller may display identification information of the food identified from the selected sample image on the touch-sensitive display.
  • the controller may display a name of a food identified from the selected sample image and the selected sample image and a storage period of the food identified from the selected sample image on the touch-sensitive display.
  • Identification information of the food identified from the selected sample image may include names of at least two foods.
  • the controller may display the at least two pieces of identification information on the touch-sensitive display.
  • the controller may display one piece of identification information selected by the user among the at least two pieces of identification information on the touch-sensitive display.
  • the controller may receive a modification of the identification information identified from the image from the user through the touch-sensitive display, and transmit the modification of the identification information identified from the image to the external device.
  • a control method of a refrigerator includes: photographing the inside of the storage room; Displaying an image inside the storage room on a touch-sensitive display; Transmit the image to an external device; Receive identification information identified from the image from the external device in response to a user's touch input to the touch sensitive display; And displaying the identification information on the touch-sensitive display.
  • the identification information may include identification information of the food identified from the image and location information in the image of the identified food.
  • the displaying of the identification information on the touch-sensitive display may include determining a food corresponding to the touch input of the user based on the location information of the user's touch input and the location information in the image of the identified food; And displaying food information corresponding to the user's touch input on the touch-sensitive display.
  • the location information in the image of the identified food may include the location of a plurality of sample images arbitrarily selected from the image.
  • the identification information of the food identified from the image may include identification information of the food identified from the plurality of sample images.
  • the determining of the food corresponding to the user's touch input may include selecting a sample image having a minimum distance from the user's touch input among the plurality of sample images.
  • Displaying the identification information on the touch-sensitive display may include displaying identification information of a food identified from the selected sample image on the touch-sensitive display.
  • the control method of the refrigerator may further include displaying a name of a food identified from the selected sample image and the selected sample image and a storage period of the food identified from the selected sample image on the touch-sensitive display.
  • the displaying of the identification information on the touch-sensitive display may further include displaying any one of the at least two pieces of identification information selected by the user on the touch-sensitive display.
  • a system includes a refrigerator; And a server device.
  • the refrigerator photographs the inside of the storage compartment, displays an image inside the storage compartment on the touch-sensitive display, transmits the image to the server device, and responds to the user's touch input to the touch-sensitive display from the server device.
  • the identification information identified from the image may be received, and the identification information may be displayed on the touch-sensitive display.
  • the server apparatus receives an image inside the storage compartment from the refrigerator, randomly selects a plurality of sample images from the image, identifies food from each of the plurality of sample images, and stores each of the plurality of sample images. Location information and identification information of the food identified from each of the plurality of sample images may be transmitted to the refrigerator.
  • the refrigerator may determine a food corresponding to the user's touch input based on the location information of the user's touch input and the location information of each of the sample images.
  • the refrigerator may select a sample image having a minimum distance from the user's touch input among the plurality of sample images.
  • the refrigerator may display identification information of the food identified from the selected sample image on the touch-sensitive display.
  • the refrigerator may display a name of the food identified from the selected sample image and the selected sample image and a storage period of the food identified from the selected sample image on the touch-sensitive display.
  • onset food management system includes a server device; And a storage compartment, and a refrigerator configured to transmit an image of the storage compartment to the server device.
  • the server device includes a communication unit; And a processing unit for identifying food from the image received from the refrigerator through the communication unit and transmitting information related to the identified food to the refrigerator through the communication unit.
  • the processor may identify food using different processes from different images.
  • the processor may identify a food region based on the difference between the received image and the reference image, and identify the food from the image of the food region.
  • the processor may identify a food by identifying the character when a character is extracted from the received image.
  • the processor may divide the received image into a plurality of areas according to a change in color of the received image, and identify food from the plurality of areas.
  • the processing unit may include a machine learning identification engine previously, and may identify food from the received image using the food food engine.
  • the refrigerator may display information related to the identified food and receive a correction input for the information related to the identified food from a user.
  • the processor may receive a correction input of the user, and may machine learn the identification engine again using the correction input of the user.
  • the refrigerator identifies the change area from a difference between the first image of the storage compartment at a first time point and the second image of the storage compartment at a second time point, and the server apparatus includes the change area to identify a food of the change area. Can be sent to.
  • a food management method includes: acquiring an image of a storage compartment by photographing an interior of the storage compartment; Identifying food from the image of the storage room using a machine learning identification engine previously; And displaying information related to the identified food. Identifying the food may include identifying food using different processes from different images.
  • Identifying the food comprises: identifying a food region based on the difference between the image and the reference image; And identifying food from the image of the food region.
  • Identifying the food may include identifying food by identifying the text when the text is extracted from the image.
  • Identifying the food comprises: dividing an image of the storage compartment into a plurality of areas according to a change in color of the image; Food can be identified from the plurality of areas.
  • the food management method may further include receiving a correction input for the information related to the identified food from a user.
  • the food management method may further include machine learning the identification engine again using a modified input of the user.
  • the food management method includes identifying a change region from a difference between a first image of the storage compartment at a first time point and a second image of the storage compartment at a second time point; The method may further include identifying a food of the modified region.
  • a refrigerator according to one aspect of the disclosed onset display; storeroom; A camera provided in the storage compartment; And a controller configured to control the camera to photograph the storage compartment, identify food stored in the storage compartment from an image of the storage compartment, and display information related to the identified food on the display device.
  • the controller may identify food using different processes from different images.
  • the control unit may include a machine learning identification engine previously, and may identify food from an image of the storage room using the food food engine.
  • the controller may identify a food region based on a difference between the image of the storage room and the reference image, and identify the food from the image of the food region.
  • the controller may identify food by identifying the character when the character is extracted from the image of the storage room.
  • the controller may divide the image of the storage compartment into a plurality of regions according to a change in the color of the image of the storage compartment, and identify food from the plurality of regions.
  • a refrigerator capable of automatically identifying food stored in the refrigerator, and a control method and system thereof.
  • a refrigerator capable of identifying food stored in the refrigerator using machine learning, a control method and a system thereof.
  • a refrigerator a control method and a system thereof, which can provide various services to a user based on food identified using machine learning.
  • FIG. 1 illustrates a food management system according to one embodiment.
  • FIG. 2 illustrates an exterior of a refrigerator according to one embodiment.
  • FIG 3 illustrates a front surface of a refrigerator according to one embodiment.
  • FIG. 4 illustrates a configuration of a refrigerator according to one embodiment.
  • FIG. 5 illustrates a touch screen display included in the refrigerator according to one embodiment.
  • FIG. 6 illustrates a configuration of a server device according to an embodiment.
  • FIG. 7 illustrates a configuration of an object identification engine included in a server device according to an embodiment.
  • FIG. 8 illustrates data stored in a server device according to an exemplary embodiment.
  • FIG. 9 illustrates an example of a learning engine for identifying an object included in a server device, according to an exemplary embodiment.
  • FIG. 10 illustrates an example of machine learning for object identification of a server apparatus according to an embodiment.
  • FIG. 11 illustrates a configuration of an object identification engine of a server device according to an embodiment.
  • FIG. 15 illustrates another example of an object identification engine of a server apparatus, according to an exemplary embodiment.
  • 16 and 17 illustrate another example in which an object identification engine included in a server device identifies an object.
  • FIG. 18 illustrates an example of a food identification method of a food management system, according to an embodiment.
  • FIG. 19 illustrates an interior image of a storage room photographed by the food identification method illustrated in FIG. 18.
  • FIG. 20 illustrates an example of identifying a food region by the food identification method illustrated in FIG. 18.
  • FIG. 21 illustrates an example of displaying food-related information by the food identification method illustrated in FIG. 18.
  • FIG. 22 illustrates another example of displaying food-related information by the food identification method illustrated in FIG. 18.
  • FIG. 23 is a view illustrating another example of a food identification method of the food management system according to one embodiment.
  • FIG. 24 illustrates an example of receiving a user's touch input by the food identification method illustrated in FIG. 23.
  • FIG. 25 illustrates an example of extracting a sample image by the food identification method illustrated in FIG. 23.
  • FIG. 26 illustrates an example of displaying food related information by the food identification method illustrated in FIG. 23.
  • FIG. 27 illustrates another example of displaying food-related information by the food identification method illustrated in FIG. 23.
  • FIG. 28 illustrates another example of a food identification method of a food management system according to one embodiment.
  • FIG. 29 illustrates an example of identifying additional food regions by the food identification method illustrated in FIG. 28.
  • FIG. 30 illustrates an example of displaying additional food related information by the food identification method illustrated in FIG. 28.
  • 31 is another example of a food identifying method of a food management system according to one embodiment.
  • 32 and 33 illustrate an example of modifying food related information by the food identification method illustrated in FIG. 31.
  • 34 is a view illustrating a method of identifying food by a refrigerator according to one embodiment.
  • FIG. 35 illustrates another example of an identification method in which the food management system individually identifies each of the foods.
  • FIG. 36 illustrates an example of manually inputting food related information by the food identification method illustrated in FIG. 35.
  • FIG. 37 illustrates another example of an identification method of collectively identifying food items by a food management system.
  • FIG. 38 illustrates an example of modifying food related information by the food identification method illustrated in FIG. 37.
  • FIG. 39 illustrates another example of modifying food related information by the food identification method illustrated in FIG. 37.
  • FIG. 40 is a view illustrating an example of a food management method of a food management system according to one embodiment.
  • FIG. 41 illustrates an example of purchasing food by the food management method illustrated in FIG. 40.
  • FIG. 42 is a view illustrating another example of a food management method of the food management system according to one embodiment.
  • 43 is a view illustrating a recipe providing method of a food management system according to one embodiment.
  • 44 and 45 illustrate an example of providing a recipe by the method of providing a recipe illustrated in FIG. 43.
  • FIG. 46 illustrates another example of providing a recipe by the recipe providing method of FIG. 43.
  • 47 is a view illustrating a recipe providing method of a food management system according to one embodiment.
  • FIG. 48 and 49 illustrate an example of providing a recipe by the method of providing a recipe illustrated in FIG. 47.
  • first, second, etc. are used to distinguish one component from another component, and the component is not limited by the terms described above.
  • the identification code is used for convenience of explanation, and the identification code does not describe the order of each step, and each step may be performed differently from the stated order unless the context clearly indicates a specific order. have.
  • FIG. 1 illustrates a food management system according to one embodiment.
  • the food management system 1 includes a refrigerator 100 for storing food and a server 200 for identifying (or recognizing) food stored in the refrigerator 100. And user equipment 300 that provides the user with information related to the identified (or perceived) grief.
  • the refrigerator 100, the server device 200, and the user device 300 included in the food management system 1 may be connected through a communication network NET.
  • the communication network (NET) may include both a wired communication network and a wireless communication network.
  • the wired communication network may include a communication network such as a cable network or a telephone network
  • the wireless communication network may include a communication network for transmitting and receiving signals through radio waves.
  • the wired communication network and the wireless communication network may be connected to each other.
  • the wired communication network may include a wide area network (WAN) such as the Internet
  • the wireless communication network may include an access point (AP) connected to the wide area network (WAN).
  • WAN wide area network
  • AP access point
  • the refrigerator 100 may refrigerate or freeze food.
  • the refrigerator 100 may include a storage compartment for storing food and a cooling device for cooling the storage compartment.
  • the refrigerator 100 may supply cooled air to the storage compartment by using a cooling device to prevent the food stored in the storage compartment from being deteriorated.
  • the refrigerator 100 may provide the user with information related to the pain stored in the storage compartment in response to the user's request. For example, the refrigerator 100 may photograph a storage room in which food is stored and acquire an image inside the storage room. In order to obtain information related to food stored in the storage room, the refrigerator 100 may transmit the captured image of the inside of the storage room to the server device 200 through a communication network NET. In addition, the refrigerator 100 may receive the food-related information (food-related information) stored in the storage room from the server device 200, display the received food-related information or transmit it to the user device 300.
  • the refrigerator 100 may photograph a storage room in which food is stored and acquire an image inside the storage room.
  • the refrigerator 100 may transmit the captured image of the inside of the storage room to the server device 200 through a communication network NET.
  • the refrigerator 100 may receive the food-related information (food-related information) stored in the storage room from the server device 200, display the received food-related information or transmit it to the user device 300.
  • the server device 200 may process data from another device through a network NET, and store or transmit the processed data to another device.
  • the server device 200 may include a communication unit capable of communicating with another device, a processing unit capable of processing data, and a storage medium capable of storing the processed data.
  • the server device 200 may be a server, a client, a workstation, a personal computer, a cloud, a data drive, a data station, or the like. May be called.
  • the server device 200 may include object recognition capable of identifying an object.
  • the server device 200 may include object recognition for identifying an object included in an image using machine learning.
  • the server device 200 may receive an image of the inside of the storage compartment from the refrigerator 100 through the communication network NET, and identify food (ie, food stored in the storage compartment) included in the image of the storage compartment.
  • the server device 200 may store information related to the identified food or transmit information related to the identified food to the refrigerator 100 and / or the user device 300.
  • the information related to the identified food may include an image of the identified food, a name of the identified food, a category of the identified food, a storage period of the identified food, and the like.
  • the user device 300 may process data from another device through a communication network NET, and display the processed data to the user.
  • the user device 300 may be carried by the user, or placed in the home or office of the user, and the user can easily access the user device 300.
  • the user device 300 may be a workstation, a personal computer, a terminal, a portable telephone, a smart phone, a handheld device, or a wearable device. May be variously named.
  • the user device 300 may receive the information related to the food stored in the refrigerator 100 from the refrigerator 100 and / or the server device 200, and display information related to the food stored in the refrigerator 100. For example, the user device 300 may display a name, an image, a category, an expiration date, and the like of the food stored in the refrigerator 100.
  • the food management system 1 may capture an image of a storage room of the refrigerator 100 to obtain an image inside the storage room, and identify a food included in the image of the storage room. In other words, the food management system 1 may identify the food stored in the storage compartment by using the image inside the storage compartment. In addition, the food management system 1 may provide the user with information related to the food stored in the storage compartment based on the identification result.
  • the refrigerator 100 may identify the food using the server device 200 and obtain information related to the stored food. have.
  • FIG. 2 illustrates an exterior of a refrigerator according to one embodiment.
  • 3 illustrates a front surface of a refrigerator according to one embodiment.
  • 4 illustrates a configuration of a refrigerator according to one embodiment.
  • 5 illustrates a touch screen display included in the refrigerator according to one embodiment.
  • the refrigerator 100 may include a main body 101 having an open front surface and a body formed inside the main body 101 and refrigerated and / or frozen.
  • the storage compartment 110 may include a door 120 that opens and closes an open front surface of the main body 101.
  • the main body 101 may form an appearance of the refrigerator 100.
  • the main body 101 may include an inner wound 102 forming the storage compartment 110 and an outer wound 103 coupled to an outer side of the inner wound 102 to form an outer appearance.
  • An insulating material (not shown) may be filled between the inner wound 102 and the outer wound 103 of the main body 101 to prevent cold air from leaking out of the storage compartment 110.
  • the storage compartment 110 may be divided into a plurality of storage compartments 110 by the horizontal barrier rib 110a and the vertical barrier rib 110b.
  • the storage compartment 110 may be divided into an upper storage compartment 111, a lower first storage compartment 112, and a lower second storage compartment 113.
  • the storage compartment 110 may be provided with a shelf 110c on which food can be placed.
  • the storage space inside the storage compartment 110 may be partitioned by the shelf 110c.
  • the upper storage room 111 may be divided into a first storage space 111a, a second storage space 111b, and a third storage space 111c.
  • the storage compartment 110 may be opened or closed by the door 120.
  • the upper storage compartment 111 may be opened and closed by the upper first door 121a and the upper second door 121b
  • the lower first storage compartment 112 may be the lower first. It may be opened and closed by the door 122
  • the lower second storage compartment 113 may be opened and closed by the lower second door 123.
  • the handle 120a may be provided in the door 120 to easily open and close the door 120.
  • the handle 121 may be formed long in the vertical direction along the upper first door 121a and the upper second door 121b and between the lower first door 122 and the lower second door 123. . Thus, when the door 120 is closed, the handle 120a can be seen as being integrally provided.
  • the refrigerator 1 includes a touch screen display 130, a temperature sensor 140, a camera 150, and a cooler as shown in FIG. 4. 160, a communicator 170, a storage 180, and a controller 190.
  • the touch screen display 130 may include a display panel 131 for displaying an image and a touch panel 132 for receiving a touch input.
  • the display panel 131 may convert image data received from the controller 190 into an optical signal that can be viewed by a user.
  • the display panel 131 may include a liquid crystal display (LCD) panel, a light emitting diode (LED) panel, an organic light emitting diode (OLED) panel, a micro light emitting diode (Micro Light Emitting Diode), uLED) panel, plasma display panel (PDP) and the like can be employed.
  • LCD liquid crystal display
  • LED light emitting diode
  • OLED organic light emitting diode
  • uLED micro light emitting diode
  • PDP plasma display panel
  • the display panel 131 is not limited thereto, and the display panel 131 may employ various displays capable of visually displaying an optical image corresponding to the image data.
  • the touch panel 132 may receive a user's touch input and output an electrical signal corresponding to the received touch input to the controller 190.
  • the touch panel 132 detects a user's touch on the touch panel 132 from a change in electrical resistance value or a change in capacitance, and controls an electric signal corresponding to the coordinates of the touch point of the user.
  • the controller 190 may identify the coordinates of the touch point of the user based on the electrical signal received from the touch panel 132.
  • the controller 190 may identify the user input based on the coordinates of the user's touch point.
  • the touch panel 132 may be located in front of the display panel 131.
  • the touch panel 132 may be provided on a surface on which an image is displayed. Therefore, the touch panel 132 may be formed of a transparent material so that the image displayed on the display panel 131 is not distorted.
  • the touch panel 132 may employ a resistive touch panel or a capacitive touch panel. However, the touch panel 132 is not limited thereto, and the touch panel 132 may detect a user's touch or approach, and output various electrical signals corresponding to the detected coordinates of the touch point or the coordinates of the access point. A touch panel can be adopted.
  • the touch screen display 130 may receive a touch input from the user, transmit the touch input to the controller 190, and display an image of the controller 190 in response to the touch input of the user.
  • the touch screen display 130 may interact with the user.
  • the touch screen display 130 may be installed in the door 120 for the user's convenience.
  • the touch screen display 130 may be installed in the upper second door 121b.
  • the touch screen display 130 installed in the upper second door 121b will be described, but the installation position of the touch screen display 130 is not limited to the upper second door 121b.
  • the touch screen display 130 may be viewed by a user such as an upper first door 121a, a lower first door 122, a lower second door 123, and an outer case 103 of the main body 10. It can be installed anywhere.
  • the touch screen display 130 may have a wake up function that is automatically activated when the user approaches within a certain range. For example, when the user approaches within a certain range, the touch screen display 130 may be activated. In other words, the touch screen display 130 may be turned on. In addition, the touch screen display 130 may be deactivated when a predetermined time elapses after the user leaves the predetermined range. In other words, the touch screen display 130 may be in an off state.
  • the temperature sensing unit 140 may be provided in the storage compartment 110 and may include inner temperature sensors 141 for sensing a temperature of the storage compartment 110.
  • the internal temperature sensors 141 are installed in each of the plurality of storage rooms 111, 112, and 112 to sense the temperature of each of the plurality of storage rooms 111, 112, and 113, and control an electrical signal corresponding to the detected temperature. And output to 190.
  • Each of the internal temperature sensors 141 may include a thermistor whose electrical resistance changes with temperature.
  • the camera 150 may be installed in the storage compartment 110 to acquire an internal image of the storage compartment 110.
  • the camera 150 may be installed inside the door 120 and may be photographed from the inside of the door 120 toward the inside of the storage compartment 110. Accordingly, the camera 150 may photograph the inside of the storage compartment 110 in a direction substantially the same as the direction of the user's gaze.
  • the camera 150 may include an imager 151 for capturing an image and converting the image into an electrical signal.
  • the imager 151 may include a plurality of photodiodes for converting an optical signal into an electrical signal, and the plurality of photodiodes may be arranged in two dimensions.
  • the imager 151 may include, for example, a charge-coupled device (CCD) image sensor or a complementary metal-oxide-semiconductor (CMOS) image sensor.
  • CCD charge-coupled device
  • CMOS complementary metal-oxide-semiconductor
  • the camera 150 may include a plurality of imagers 151a, 151b, and 151c to photograph the interior of the storage compartment 110 partitioned by the shelf 110c.
  • the camera 150 includes a first imager 151a for photographing the first storage space 111a of the upper storage room 111 and a second imager 151b for photographing the second storage space 111b.
  • a third imager 151c for photographing the third storage space 111c.
  • the internal image of the storage compartment 110 photographed by the camera 150 may include an image of the food stored in the storage compartment 110.
  • the camera 150 may transmit the captured image to the controller 190.
  • the camera 150 may transmit the internal image of the storage compartment 110 including the image of the food to the controller 190.
  • the cooling unit 160 may supply cold air to the storage compartment 110. Specifically, the cooling unit 160 may maintain the temperature of the storage compartment 110 within a range specified by the user by using evaporation of the refrigerant.
  • the cooling unit 160 includes a compressor 161 for compressing the refrigerant in the gas state, a condenser 162 for converting the compressed gas refrigerant into a liquid state, an expander 163 for reducing the refrigerant in the liquid state, and
  • the vaporizer may include an evaporator 164 converting the refrigerant in a reduced pressure state into a gas state.
  • the cooling unit 160 may cool the air in the storage chamber 110 by using a phenomenon in which the liquid refrigerant absorbs thermal energy of ambient air while converting the state into a gas state.
  • the cooling unit 160 is not limited to including the compressor 161, the condenser 162, the expander 163, and the evaporator 164.
  • the cooling unit 160 may include a Peltier element using the Peltier effect.
  • the Peltier effect means that when a current flows through contact surfaces of different metals, an exothermic phenomenon occurs in one metal and an endothermic phenomenon occurs in another metal.
  • the cooling unit 160 may cool the air in the storage compartment 110 using the Peltier element.
  • the cooling unit 160 may include a magnetic cooling device using a magneto-caloric effect.
  • the magnetocaloric effect means that heat is released when a specific material (magnetic calorie material) is magnetized, and absorbs heat when the specific material (magnetic calorie material) is demagnetized.
  • the cooling unit 160 may cool the air in the storage compartment 110 by using a magnetic cooling device.
  • the communication unit 170 may exchange data with the server device 200 and / or the user device 300.
  • the communicator 170 may transmit an image of the storage room captured by the camera 150 to the server device 200, and receive information related to food included in the image of the storage room from the server device 200. Can be.
  • the communication unit 170 wirelessly communicates data with the server device 200 and / or the user device 300 in a wired communication module 172 and the server device 200 and / or the user device 300. It may include a wireless communication module 171 to send and receive.
  • the wired communication module 172 may be connected to the wired communication network and communicate with the server device 200 and / or the user device 300 through the wired communication network.
  • the wired communication module 172 connects to a wired communication network through Ethernet (Ethernet, IEEE 802.3 technical standard), and receives data from the server device 200 and / or the user device 300 through the wired communication network. Can be.
  • Ethernet Ethernet, IEEE 802.3 technical standard
  • the wireless communication module 171 may wirelessly communicate with a base station or an access point (AP), and may access a wired communication network through a base station or an access point.
  • the wireless communication module 171 may also communicate with a server device 200 and / or a user device 300 connected to a wired communication network via a base station or an access point.
  • the wireless communication module 171 wirelessly communicates with an access point (AP) using Wi-Fi (WiFi TM, IEEE 802.11 technology standard), or CDMA, WCDMA, GSM, Long Term Evolution (LET), WiBro, etc. It can communicate with the base station using.
  • the wireless communication module 171 may also receive data from the server device 200 and / or the user device 300 via a base station or access point.
  • the wireless communication module 171 may communicate directly with the server device 200 and / or the user device 300.
  • the wireless communication module 171 may use the Wi-Fi, Bluetooth (Bluetooth TM, IEEE 802.15.1 technology standard), ZigBee (ZigBee TM, IEEE 802.15.4 technology standard), etc.
  • server device 200 and / or Data may be wirelessly received from the user device 300.
  • the communication unit 170 may exchange data with the server device 200 and / or the user device 300, and control the data received from the server device 200 and / or the user device 300.
  • the storage unit 180 may include a storage medium 181 for storing a program and / or data, and a connection terminal 182 that may be connected to an external storage medium for storing a program and / or data.
  • the program includes a plurality of instructions combined to perform a specific function, and data may be processed and / or processed by the plurality of instructions included in the program.
  • the programs and / or data may include system programs and / or system data directly related to the operation of the refrigerator 100, and application programs and / or application data that provide convenience and fun to the user.
  • the storage medium 181 may store a program and / or data in the form of a file.
  • the storage medium 181 may store a program and / or data in the form of a file such as "* .exe” or "* .jpg” or "* .mpg”.
  • the storage medium 181 can store the content data electrically, magnetically or optically.
  • the storage medium 181 may include a solid state driver (SSD), a hard disc drive (HDD), an optical disc drive (ODD), or the like.
  • SSD solid state driver
  • HDD hard disc drive
  • ODD optical disc drive
  • the storage medium 181 may output a program and / or data to the controller 190 in response to a loading command of the controller 190.
  • connection terminal 182 may be connected to an external storage medium storing a program and / or data.
  • the connection terminal 182 may include a universal serial bus (USB) terminal, and may be connected to an external USB storage medium.
  • USB universal serial bus
  • the external storage medium may store a program and / or data in the form of a file, and may store the program and / or data electrically, magnetically, or optically.
  • the external storage medium may output a program and / or data to the controller 190 through the connection terminal 182 in response to a loading command of the controller 190.
  • the storage unit 180 may store a program and / or data, and output the stored program and / or data to the controller 190.
  • the storage unit 180 may store a program and / or data executed by the controller 190 to perform an operation described below.
  • the controller 190 may include a memory 192 for loading and storing a program and / or data stored in the storage 180, and a user of the touch screen display 130 according to the program and / or data stored in the memory 192. It includes a processor 191 for processing the input data and the communication data of the communication unit 170.
  • the controller 190 may further include not only hardware such as the memory 192 and the processor 191, but also software such as a program and / or data stored in the memory 192 and processed by the processor 191.
  • the memory 192 may store programs and / or data for controlling the components included in the refrigerator 100, and may store temporary data generated while controlling the components included in the refrigerator 100.
  • the memory 192 may store a program and / or data for controlling the operation of the cooling unit 160 based on the sensing result of the temperature sensing unit 140, and the sensing of the temperature sensing unit 140. You can temporarily remember the result.
  • the memory 192 may store a program and / or data for controlling the operation of the touch screen display 130, and may display an image displayed on the touch screen display 130 and a touch input input through the touch screen display 130. User input) can be temporarily stored.
  • the memory 192 may store a program and / or data for controlling the photographing operation of the camera 150 based on the opening and closing of the door 120, and may temporarily store an image photographed by the camera 150. .
  • the memory 192 may store a program and / or data for controlling a communication operation of the communication unit 170, and may temporarily store communication data transmitted and received by the communication unit 170. In addition, the memory 192 may store a program and / or data for the refrigerator 100 to perform an operation described below.
  • the memory 192 is a nonvolatile memory such as read only memory or flash memory for storing data for a long time, and a static random access memory (S-RAM) for temporarily storing data. ), And volatile memory such as D-RAM (Dynamic Random Access Memory).
  • the processor 191 processes the touch input of the touch screen display 130 and / or the communication data of the communication unit 170 according to the program and / or data stored / stored in the memory 192, and the camera 150 and the touch screen.
  • a control signal for controlling the operation of the display 130 and / or the operation of the communication unit 170 may be generated.
  • the processor 191 receives temperature information of the storage compartment 110 from the temperature sensing unit 140, and controls cooling to control an operation of the cooling unit 160 based on the temperature information of the storage compartment 110. You can generate a signal.
  • the processor 191 receives a user's touch input from the touch screen display 130, and displays a display control signal and image data for displaying an image on the touch screen display 130 in response to the user's touch input. 130).
  • the processor 191 generates a shooting control signal so that the camera 150 photographs the interior of the storage room 110 as soon as the open door 120 is closed, and receives an internal image of the storage room 110 from the camera 150. can do.
  • the processor 191 may transmit a communication control signal for controlling the communication unit 170 to transmit the internal image of the storage compartment 110 to the server device 200 in response to the reception of the internal image of the storage compartment 110 from the camera 150. Can be generated.
  • the processor 191 may display the food related information on the touch screen display 130 in response to receiving the information related to the food stored in the storage compartment 110 from the server device 200 through the communication unit 170. Can be generated.
  • the processor 191 may generate a control signal for the refrigerator 100 to perform an operation described below.
  • the processor 191 may include a core for performing logical and arithmetic operations, a register for storing the calculated data, and the like.
  • the controller 190 is based on the temperature of the storage compartment 110 through the temperature sensor 140, the user's touch input through the touch screen display 130 and / or communication data through the communication unit 170 based on the cooling unit ( 160, the touch screen display 130, and the communicator 170 may be controlled.
  • the controller 190 may include a processor and a memory for controlling all of the components included in the refrigerator 100, and the controller 190 may include a plurality of processors and a plurality of memories. Can be.
  • the controller 190 may include a processor and memory for controlling the operation of the temperature sensing unit 140 / cooling unit 160 and a processor and memory for controlling the operation of the touch screen display 130. A processor and a memory for controlling the operation may be provided separately.
  • FIG. 6 illustrates a configuration of a server device according to an embodiment.
  • 7 illustrates a configuration of an object identification engine included in a server device according to an embodiment.
  • 8 illustrates data stored in a server device according to an exemplary embodiment.
  • 9 illustrates an example of a learning engine for identifying an object included in a server device, according to an exemplary embodiment.
  • 10 illustrates an example of machine learning for object identification of a server apparatus according to an embodiment.
  • the server apparatus 200 includes an input / output unit 210, a communication unit 220, a storage unit 230, and a processing unit 240. can do.
  • the input / output unit 210 may include an input device that receives a control command for the server device 200 from a user and / or an administrator, and a display device that displays operation information of the server device 200.
  • the input / output unit 210 may include various buttons or switches, pedals, keyboards, mice, track-balls, various levers, and handles for input of a user and / or an administrator.
  • a graphical user interface i.e., software input device 211, such as a hardware input device 211, such as a stick or the like, or a touch pad for input of a user and / or administrator. It may be.
  • the input / output unit 210 may include a display 212 for displaying the operation of the server device 200.
  • the input / output unit 210 may include a terminal provided separately from the server device 200.
  • the input / output unit 210 may include a fixed terminal including a keyboard, a monitor, or the like, or a portable terminal including a keyboard, a touch pad, a display, a touch screen display, and the like.
  • the communicator 220 may exchange data with the refrigerator 100 and / or the user device 300.
  • the communication unit 220 may receive an image of the inside of the storage room from the refrigerator 100, and transmit information related to food included in the image of the storage room to the refrigerator 100 and / or the user device 300. .
  • the communication unit 220 wirelessly exchanges data with the refrigerator 100 and / or the user device 300 by wired communication module 221 that exchanges data with the refrigerator 100 and / or the user device 300 by wire.
  • the receiving wireless communication module 222 may be included. Configuration and operation of the wired communication module 221 and the wireless communication module 222 may be the same as that of the refrigerator 100.
  • the communication unit 220 may exchange data with the refrigerator 100 and / or the user device 300, and output the data received from the refrigerator 100 and / or the user device 300 to the processor 240. can do.
  • the storage unit 230 may store a program and / or data for processing a request of the refrigerator 100 and / or the user device 300.
  • the storage 230 may use an object identification engine 231 for identifying an object from an image, training data 232 for training the object identification engine 231, and an object identification engine 231. It may include user modification data 233 for retraining the object identification engine 231 according to the modification of.
  • the object identification engine 231 may identify the object from the image using learning by map or learning without guidance. For example, the object identification engine 231 may identify foods included in the image and output food-related information such as names of the identified foods by pre-training or post-training.
  • the object identification engine 231 includes supervised learning that is learned by a manager's guidance, unsupervised learning that is learned without a manager's guidance, and reinforcement learning that is learned by compensation without a manager's guidance. And the like. In the following description, it is assumed that the object identification engine 231 includes supervised learning.
  • the object identification engine 231 may include various learning algorithms for object identification through an image.
  • the object identification engine 231 may include a neural network algorithm, a support vector machine (SVM) algorithm, an adaboost algorithm, a random forest algorithm, and the like. have.
  • SVM support vector machine
  • the object identification engine 231 may be trained in advance by the training data 232. In addition, the object identification engine 231 may be post-trained later by the user modification data 233.
  • the training data 232 may include an image and a name of an object corresponding to the image.
  • the training data 232 may include a file containing an “apple image” and a food name corresponding to “apple”.
  • the user modification data 233 may include an image of the object and the name of the object modified by the user of the refrigerator 100.
  • the user correction data 233 may include a file containing an “apple image” and a food name “apple” modified by a user.
  • the storage unit 230 may include a food database 232a and a user input database 233a.
  • the food database 232a may include training data 232 for training the object identification engine 231.
  • the training data 232 of the food database 232a may be stored in advance by the designer, and initially the object identification engine 231 may be trained by the training data 232 of the food database 232a. .
  • the training data 232 of the food database 232a may include a name of the food and images of the food.
  • the food database 232a may include pineapples and at least one image representing pineapples (pineapple_1.jpg, pineapple_2.jpg), grapes and grapes. It may include at least one image (Grape_1.jpg, Grapge_2.jpg) indicating, and may include an apple and at least one image (Apple_1) indicating the apple.
  • the user input database 233a may include user correction data 233 input or modified by the user.
  • the object identification engine 231 may be retrained by the user modification data 233.
  • the user modification data 233 of the user input database 233a may include images corresponding to the name of the food input by the user and the name of the modified food by the user.
  • the user input database 233a may include an apple input by the user and an image (Apple_2.jpg) corresponding to the modified apple.
  • the storage unit 230 may include a training data generator 231a, The object identification engine learner 231b and the object identification engine generator 231c may be further included.
  • the training data generator 231a may generate training data for retraining the object identification engine 231 using the existing training data 232 and the user modification data 233 input by the user. For example, the training data generator 231a may mix the training data 232 and the user correction data 233, and adjust the ratio between the training data 232 and the user correction data 233.
  • the object identification engine learner 231b may train the object identification engine 231 using training data newly generated by the training data generator 231a. As described below, the object identification engine learning unit 231b inputs a food image into the object identification engine 231 among the training data, and outputs a food name corresponding to the food image and the food output from the object identification engine 231. The names may be compared, and coefficients included in the object identification engine 231 may be adjusted according to the comparison result.
  • the object identification engine generator 231c may generate the object identification engine 231 trained by the object identification engine learner 231b.
  • the storage unit 230 may store a program and / or data, and output the stored program and / or data to the processor 240.
  • the storage 230 may store a program and / or data executed by the processor 240 to perform an operation described below.
  • the storage unit 230 may store training data 232 and user modification data 233 for training the object identification engine 231 learning engine for object identification.
  • the processor 240 may load and store a program and / or data stored in the storage 230 and the refrigerator 100 and / or a user according to the program and / or data stored in the memory 242. It may include a processor 241 for processing data received from the device 300.
  • the processor 240 may further include not only hardware such as the memory 242 and the processor 241, but also software such as programs and / or data stored in the memory 242 and processed by the processor 241.
  • the memory 242 stores programs and / or data for processing data received from the refrigerator 100 and / or the user device 300, and the data received from the refrigerator 100 and / or the user device 300. You can store temporary data that occurs during processing.
  • the memory 242 may store the untrained object identification engine 231, and train the untrained object identification engine 231 using the training data 232 stored in the storage 230. Can store programs and / or data.
  • the memory 242 may store the trained object identification engine 231, and may store a program and / or data for identifying an object of the image from the image data using the trained object identification engine 231.
  • the memory 242 may also store programs and / or data for the server device 200 to perform the operations described below.
  • the memory 242 may include a nonvolatile memory such as a ROM or a flash memory for storing data for a long time, and a volatile memory such as an S-RAM or a D-RAM for temporarily storing the data.
  • a nonvolatile memory such as a ROM or a flash memory for storing data for a long time
  • a volatile memory such as an S-RAM or a D-RAM for temporarily storing the data.
  • the processor 241 processes data received from the refrigerator 100 and / or the user device 300 according to a program and / or data stored / stored in the memory 242, and returns the processed result to the refrigerator 100 and / or.
  • a control signal for transmitting to the user device 300 may be generated.
  • the processor 241 may train the untrained object identification engine 231 stored in the memory 242. Specifically, the processor 241 inputs the training data 232 to the untrained object identification engine 231, compares the output of the object identification engine 231 with the training data 232, and then executes the object identification engine 231. Can be updated (trained).
  • the processor 241 may identify the object from the image using the trained object identification engine 231 stored in the memory 242.
  • the processor 241 inputs a storage interior image transmitted from the refrigerator 100 to the trained object identification engine 231, and displays foods including the storage interior image based on the output of the object identification engine 231. Can be identified.
  • the processor 241 may identify the food items stored in the storage compartment 110 of the refrigerator 100 using the trained object identification engine 231 and output information about the identified food items.
  • the processor 241 may generate a control signal for the server device 200 to perform the operation described below.
  • the processor 241 may include a core for performing logical and arithmetic operations, a register for storing the calculated data, and the like.
  • the processor 240 may train the untrained object identification engine 231, and identify the object from the input image using the trained object identification engine 231.
  • the object identification engine 231 may include a neural network as shown in FIG.
  • the object identification engine 231 may include an input layer into which image data is input, an output layer in which information about the identified object is output, and a hidden layer between the input layer and the output layer. (hidden) may be included.
  • the input layer may include a plurality of input nodes i1 and i2, and image data may be input to each of the plurality of input nodes i1 and i2. For example, luminance values and / or color values of the plurality of pixels included in the image may be input to each of the plurality of input nodes i1 and i2.
  • the hidden layer includes a plurality of hidden nodes h1 and h2, and a plurality of input nodes i1 and i2 to which weights w1, w2, w3 and w4 are applied to each of the hidden nodes h1 and h2.
  • the sum of the output values of) may be input.
  • the processor 240 may calculate a sum of an output value of the first input node i1 to which the first weight w1 is applied and an output value of the second input node i2 to which the second weight w2 is applied. 1 can be input to the hidden node (h1).
  • the processor 240 hides the sum of the output value of the first input node i1 to which the third weight w3 is applied and the output value of the second input node i2 to which the fourth weight w4 is applied to the second hidden value. Can input to node h2.
  • the processor 240 may apply a step function or a sigmoid function to an input value input to the hidden layer. For example, the processor 240 may input an input value of the first hidden node h1 to the sigmoid function, and output an output value of the sigmoid function to the output layer. In addition, the processor 240 inputs the input value of the second hidden node h2 to the sigmoid function and applies the weights w5, w6, w7, and w8 to the output value of the sigmoid function to output the output layer ( output).
  • the output layer includes a plurality of output nodes o1 and o2, and each of the plurality of output nodes o1 and o2 has a plurality of hidden nodes h1, to which weights w5, w6, w7 and w8 are applied.
  • the sum of the output value of h2) can be input.
  • the processor 240 may calculate a sum of an output value of the first hidden node h1 to which the fifth weight w5 is applied and an output value of the second hidden node h2 to which the sixth weight w6 is applied. It can be input to one output node o1.
  • the processor 240 outputs a sum of an output value of the first hidden node h1 to which the seventh weight w7 is applied and an output value of the second hidden node h2 to which the eighth weight w8 is applied to the second output. Can input to node o2.
  • the processor 240 may apply a step function or a sigmoid function to an input value input to an output layer. For example, the processor 240 may input an input value of the first output node o1 to the sigmoid function and output an output value of the sigmoid function. In addition, the processor 240 may input an input value of the second output node o2 to the sigmoid function and output an output value of the sigmoid function.
  • Objects are assigned to each of the plurality of output nodes o1 and o2 included in the output layer, and the output values of the plurality of output nodes o1 and o2 are displayed in the plurality of output nodes o1 and o2. It can represent the probability that the object is assigned to).
  • the processor 240 may identify an object included in the image based on the outputs of the plurality of output nodes o1 and o2.
  • the output of the object identification engine 231 may vary according to the value of the weights w1 to w8 between the layers input, hidden, and output. Accordingly, the processor 240 may set appropriate weights w1 to w8 for accurately identifying the object, and set appropriate weights w1 to w8 of the object identification engine 231 using the training data 232. have. As such, training the object identification engine 231 using the training data 232 is referred to as "learning.”
  • the server device 200 may train the untrained object identification engine 231 as shown in FIG. 10A.
  • the server device 200 obtains training data 232 (1010).
  • the processor 240 may load the training data 232 stored in the storage 230.
  • the training data 232 may include a plurality of images and names of objects corresponding to the plurality of images, respectively.
  • the server device 200 may obtain training data 232 from another device connected to a wide area network (WAN).
  • WAN wide area network
  • the server device 200 inputs the training data 232 to the object identification engine 231 (1020).
  • the processor 240 may determine luminance values and / or RGB values (data of a red subpixel, data of a green subpixel, and blue subpixel) of a plurality of pixels constituting an image of the training data 232. 231). For example, the processor 240 may input RGB values of a plurality of pixels constituting the image of the training data 232 to the input nodes i1 and i2 of the input layer of the neural network.
  • the server device 200 compares the training data 232 with the output of the object identification engine 231 (1030).
  • the processor 240 performs object identification on an image of the training data 232 using the object identification engine 231. For example, the processor 240 may apply the weights w1 to w4 to the outputs of the input nodes i1 and i2 of the input layer of the neural network, thereby applying the hidden nodes h1, of the hidden layer. h2). The processor 240 inputs the inputs of the hidden nodes h1 and h2 to the sigmoid function and applies the weights w5-w8 to the output of the sigmoid function to output nodes of the output layer (ouput) ( o1, o2). Thereafter, the processor 240 may input the inputs of the output nodes o1 and o2 to the sigmoid function and output the output of the sigmoid function.
  • the processor 240 may identify an object included in the image of the training data 232 based on the output of the object identification engine 231. For example, each of the plurality of output nodes o1 and o2 of the neural network may output a probability that an object included in the image matches objects allocated to each of the plurality of output nodes o1 and o2. The processor 240 may identify the object included in the image of the training data 232 based on the probability output from the neural network.
  • the processor 240 may compare the object identified by the output of the object identification engine 231 with the object by the training data 232.
  • the server device 200 updates the object identification engine 231 based on the output of the object identification engine 231 and the training data 232 (1040).
  • the processor 240 may determine whether the object identified by the output of the object identification engine 231 and the object by the training data 232 are the same.
  • the processor 240 updates the object identification engine 231 in an appropriate manner. For example, if the probability output from the neural network for the object by the training data 232 is less than the reference value, the processor 240 may change the weights w1-w8 constituting the neural network.
  • the server device 200 may repeat the acquisition of the training data 232, the input of the training data 232, the evaluation of the output of the object identification engine 231, and the update of the object identification engine 231.
  • the object identification engine 231 is trained to identify an object included in the image of the training data 232.
  • the weights w1-w8 of the neural network are set to identify objects included in the image of the training data 232 by the training operation.
  • server device 200 may identify the object of the image using the trained object identification engine 231 as shown in (b) of FIG. 10.
  • the server device 200 receives an image (1060).
  • the processor 240 may receive an image including an object to be identified from an external device through the communication unit 220.
  • the processor 240 may receive an internal image of the storage compartment 110 from the refrigerator 100 through the communication unit 220.
  • the server device 200 inputs the received image to the object identification engine 231 (S1070).
  • the processor 240 may input luminance values and / or RGB values (data of a red subpixel, data of a green subpixel, and data of a blue subpixel) of a plurality of pixels constituting the image to the object identification engine 231. have. Operation 1070 may be the same as operation 1020.
  • the server device 200 performs object identification (1080).
  • the processor 240 may recognize the object in the received image using the object identification engine 231, and identify the object included in the image of the training data 232 based on the output of the object identification engine 231. have. Operation 1080 may be the same as operation 1030.
  • the server device 200 outputs an object identification result (1090).
  • the processor 240 may control the communicator 220 to transmit information about the object of the received image to another device. For example, the processor 240 may transmit the information related to the food identified from the internal image of the storage compartment 110 to the refrigerator 100 through the communicator 220.
  • the server device 200 may identify the food using a special method in order to identify the food stored in the storage compartment 110 of the refrigerator 100.
  • 11 illustrates a configuration of an object identification engine of a server device according to an embodiment.
  • 12, 13, and 14 illustrate an example in which the object identification engine included in the server device identifies food.
  • the server device 200 may include an area detection engine 251, an area classification engine 252, and an identification engine 253.
  • the area detection engine 251, the area classification engine 252, and the identification engine 253 may be implemented in hardware or software, respectively.
  • the area detection engine 251, the area classification engine 252, and the identification engine 253 may be part of the processor 241 in hardware, or may be part of a program stored in the storage unit 230 in software.
  • the area detection engine 251 acquires an interior image 500 of the storage compartment 110 of the refrigerator 100, and positions foods 501, 502, 503, and 504 in the storage interior image 500.
  • the food regions 510, 520, and 530 may be identified.
  • the area detection engine 251 may identify food areas 510, 520, and 530 in which food is located in various ways.
  • the area detection engine 251 selects the food areas 510, 520, 530 and the background area based on the difference between the internal image of the empty storage compartment 110 and the internal image of the storage compartment 110 in which the food is placed. Can be separated.
  • the area detection engine 251 may receive both a storage room interior image and a reference image (empty storage room interior image) from the refrigerator 100, and the area detection engine 251 may locate a food from a difference between the storage room interior image and the reference image.
  • the food regions 510, 520, and 530 may be separated from the background region serving as the background of the storage compartment 110.
  • the area detection engine 251 may extract an edge from an image of the inside of the storage room, and separate the background area from the food areas 510, 520, and 530 where the food is located based on the image's outline. have.
  • the area detection engine 251 extracts information about a change in color from an image of the inside of the storage room, and the food areas 510 and 520 where the food is located based on a boundary line in which the color rapidly changes within the image inside the storage room. 530 may be separated from the background region.
  • the area detection engine 251 may remove an image of the shelf 110c and / or a reflection image of the food from the food areas 510, 520, and 530 in order to improve the identification rate of the food.
  • the area detection engine 251 may acquire a storage interior image 500 as illustrated in FIG. 12A, and may be obtained from the storage interior image 500 as illustrated in FIG. 12B. As shown, food regions 510, 520, and 530 in which foods 501, 502, 503, and 504 are located may be identified.
  • the region detection engine 251 includes a first food region 510 in which the first food A 501 is located, a second food region 520 in which the second food B 502 is located, and a third food C ( Third food region 530 in which 503 and fourth food D 504 are located together may be identified.
  • the area classification engine 252 may classify the food areas 510, 520, 530 identified by the area detection engine 251. For example, the area classification engine 252 may classify the food areas 510, 520, 530 according to the difficulty (degree of difficulty) for identifying the food from each of the food areas 510, 520, 530. .
  • the area classification engine 252 may include a character identification area in which characters are detected in the food areas 510, 520, and 530, an image identification area in which a single food is detected in the food areas 510, 520, and 530, and In the regions 510, 520, and 530, the food may be classified into an image segmentation area in which a plurality of foods are detected.
  • the area classification engine 252 may classify it into a character identification region. For example, processed foods such as canned food and ham do not have an inherent shape for identifying food, and letters for identifying food are provided in the appearance of the food. Food in the letter identification area can be identified by letter identification.
  • the area classification engine 252 uses the edge extraction and / or color change extraction for the food regions 510, 520, and 530. In operation 530, it may be determined whether a single food item or a plurality of food items are overlapped with each other.
  • the area classification engine 252 locates a plurality of foods overlapping each other in the food area 510, 520, and 530. Can be classified into an image segmentation area.
  • the area classification engine 252 may display an image in which a single food is located in the food areas 510, 520, and 530. You can classify it into a human-eye area
  • the area classification engine 252 may extract the letter “A” from the first food area 510 shown in FIG. 12B using a letter extraction algorithm. Therefore, the area classification engine 252 may classify the first food area 510 as a letter identification area.
  • the area classification engine 252 is unable to extract characters from the second food area 520 shown in FIG. 12 (b), and finds a boundary line in which the color suddenly changes within the second food area 520. Can not. Therefore, the area classification engine 252 may classify the second food area 520 as an image identification area.
  • the area classification engine 252 may find a boundary line in which the color rapidly changes within the third food area 530 illustrated in FIG. 12B. Therefore, the area classification engine 252 may classify the third food area 530 as an image segmentation area.
  • the identification engine 253 uses the different food identification methods (or identification engines) for the food areas 510, 520, 530 classified by the area classification engine 252, and the food areas 510, 520, 530. ) Foods can be identified. Specifically, the identification engine 253 uses the at least one of the letter identification engine 253a, the image identification engine 253b, and the image segmentation engine 253c with respect to the food regions 510, 520, and 530. Food 510, 520, 530 can be identified.
  • the identification engine 253 may use the character identification engine 253a to identify the letter "A" from the first food region 510 as shown in FIG. Food A 501 of the first food region 510 may be identified based on A ′′.
  • the identification engine 253 may identify the food B 502 from the second food region 520 as shown in FIG. 12C using the image identification engine 253b.
  • the image identification engine 253b may identify the food B 502 from the image itself of the second food region 520.
  • the identification engine 253 may segment the images of the foods superimposed on each other using the region division engine 253c.
  • the region segmentation engine 253c may convert the image segmentation region into a color space in which the plurality of foods are distinguished, and then segment the image segmentation region into a plurality of divided food regions based on an appropriate threshold value.
  • the region segmentation engine 253c may group regions having similar colors as the image segment region and divide the region into a plurality of divided food regions.
  • the third food region 530 illustrated in FIG. 12B is divided into the first divided food region 531 and the second divided food region as illustrated in FIG. 12C. 532).
  • the identification engine 253 may perform food identification on each of the first divided food region 531 and the second divided food region 532 using the image identification engine 253c.
  • the identification engine 253 may identify food C 503 from the first divided food region 531, and may identify food D 504 from the second divided food region 532.
  • the area detection engine 251 may identify the food areas 510, 520, and 530 on which foods are located by comparing the image of the empty storage room 110 with the received image.
  • the area classification engine 252 may classify the food areas 510, 520, and 530 based on whether the food areas 510, 520, and 530 include text and / or color boundaries.
  • the identification engine 253 may identify foods using different identification algorithms for the food areas 510, 520, and 530 classified into different groups. For example, the identification engine 253 uses the character identification engine 253a for the food region including the character to identify the food in the food region, and the region segmentation engine 253c for the food region including the color boundary. The food region can be divided by using. In addition, the identification engine 253 may identify the food from the image using the image identification engine 253b.
  • the server device 200 may output a storage room interior image 400 including information about foods 410, 402, 403, and 404 from the storage interior image 400. .
  • the area detection engine 251 may acquire an interior image 400 of the storage compartment, the first food region 410 in which the first food 401 is located from the interior of the storage compartment 400, and the second food 402.
  • the second food region 420 in which is located and the third food region 430 in which the third food 403 and the fourth food 404 are located together may be identified.
  • the region classification engine 252 may classify the first food region 410 from which the text is extracted into the character identification region, and classify the second food region 420 from which the character is not extracted into the image identification region.
  • the third food region 430 including a boundary line in which the image changes rapidly may be classified as an image segmentation region.
  • the character identification engine 253a of the identification engine 253 may identify the first food 401 of the first food region 410 by using character identification, and the image identification engine 253b of the identification engine 253. May identify the second food 402 of the second food region 420 by using the image identification.
  • the region dividing engine 253c of the identification engine 253 may divide the third food region 430 into an image of the third food 403 and an image of the fourth food 404, and the character identification engine 253a.
  • the image identification engine 253b may identify the third food 403 and the fourth food 404 from the image of the third food 403 and the image of the fourth food 404, respectively.
  • the identification engine 253 may tag the first food 401 in the first food region 410, may tag the second food 402 in the second food region 420, and may include the third food region.
  • the third food 403 and the fourth food 404 may be tagged 430.
  • the identification engine 253 may include the first food region 410 tagged with the first food 401, the second food region 420 tagged with the second food 402, and the third food 403 and the third food. 4
  • the food 404 may output a storage room interior image 400 including the third food region 430 tagged therein.
  • the server device 200 may be added to the new storage room interior image 450 by using the storage interior image 400 in which foods 401, 402, 403, and 404 have been identified in advance. Food 461 can be identified.
  • the area detection engine 251 may acquire a new storage interior image 450, and based on the difference between the existing storage interior region 400 and the new storage interior image 450, the fifth food 461 may be located.
  • the fourth food region 460 can be identified.
  • the area classification engine 252 may classify the fourth food area 460 into any one of a character identification area, an image identification area, and an image segmentation area.
  • the identification engine 253 may identify the fifth food 461 of the fourth food area 460 according to the classification (character identification area, image identification area, or image segmentation area) of the fourth food area 460. Can be.
  • the identification engine 253 may tag the fifth food 461 in the fourth food area 460.
  • the food identification engine 253 combines the fourth food region 460 in which the fifth food 461 is tagged with the image 400 of the existing storage compartment, and the new storage compartment in which the fifth food 461 is additionally tagged.
  • the internal image 450 may be output.
  • 15 illustrates another example of an object identification engine of a server apparatus, according to an exemplary embodiment.
  • 16 and 17 illustrate another example in which an object identification engine included in a server device identifies an object.
  • the server device 200 may include an image sampling engine 261 and an image identification engine 262.
  • the image sampling engine 261 and the image identification engine 262 may be implemented in hardware or software, respectively.
  • the image sampling engine 261 and the image identification engine 262 may be part of the processor 241 in hardware, or may be part of a program stored in the storage unit 230 in software.
  • the image sampling engine 261 may acquire a storage interior image 500 photographing the interior of the storage chamber 110 of the refrigerator 100, and extract a plurality of sample images 541 to 546 from the storage interior image 500. Can be.
  • the image sampling engine 261 may extract the sample images 541 to 546 in various ways.
  • the image sampling engine 261 may extract sample images 541 to 546 having an arbitrary size (horizontal length, vertical length) at an arbitrary position (x-axis coordinate, y-axis coordinate).
  • the image sampling engine 261 may include a random function, and selects the x-axis coordinate (x), the y-axis coordinate (y), the horizontal length (h), and the vertical length (v) of the sample image by using the random function. Can be.
  • the image sampling engine 261 may extract an image having the selected coordinates (x, y) and sizes (h, v) from the storage interior image 500.
  • the image sampling engine 261 may select any first coordinates x1 and y1 and an arbitrary first size h1 and v1, and the first coordinates x1.
  • a first sample image 541 having a y1 and a first size h1 and v1 may be extracted from the storage interior image 500.
  • the image sampling engine 261 may include a second sample image 542 having a second coordinate (x2, y2) and a second size (h2, v2), a third coordinate (x3, y3), and a third size (h3).
  • An n th sample image having a size may be extracted.
  • the number of sample images is not limited.
  • the image sampling engine 261 may extract a plurality of sample images having a predetermined size at a predetermined position.
  • the image sampling engine 261 may partition the storage interior image 500 into a predetermined size, and may acquire sample images partitioned from the storage interior image 500.
  • the image sampling engine 261 may acquire sample images having a predetermined size from the storage interior image 500 at predetermined intervals.
  • the image sampling engine 261 may extract a plurality of sample images having a predetermined size at an arbitrary position.
  • the image identification engine 262 may identify foods included in the storage interior image 500 based on the plurality of sample images 541 to 546 extracted by the image sampling engine 261.
  • Each of the plurality of sample images 541 to 546 is provided to the image identification engine 262, and the image identification engine 262 may identify foods included in each of the plurality of sample images 541 to 546.
  • Image identification engine 262 may include, for example, a neural network. Luminance values and / or color values of a plurality of pixels included in the sample image are input to each of the plurality of input nodes i1 and i2 of the neural network, and the plurality of output nodes o1 and o2 are included in the sample image. It is possible to output a numerical value (eg, a probability) indicating which food among the plurality of predetermined foods.
  • a numerical value eg, a probability
  • the image identification engine 262 may output the first identification result and the second identification result based on the numerical value output from the neural network. For example, the image identification engine 262 may select a food having the largest numerical value output from the neural network as the first identification result, and select a food having the second largest numerical value output from the neural network as the second identification result. have.
  • the image identification engine 262 may output identification data including coordinates and sizes indicating a sample image, a first identification result of the sample image, and a second identification result of the sample image.
  • the identification data includes coordinates (x1, y1) and size (h1, v1) of the first sample image 541, a first identification result, a second identification result, and a second sample image.
  • the operation of the food management system 1 including the refrigerator 100, the server device 200, and the user device 300 will be described.
  • FIG. 18 illustrates an example of a food identification method of a food management system, according to an embodiment.
  • FIG. 19 illustrates an interior image of a storage room photographed by the food identification method illustrated in FIG. 18.
  • 20 illustrates an example of identifying a food region by the food identification method illustrated in FIG. 18.
  • FIG. 21 illustrates an example of displaying food-related information by the food identification method illustrated in FIG. 18.
  • FIG. 22 illustrates another example of displaying food-related information by the food identification method illustrated in FIG. 18.
  • the refrigerator 100 acquires an internal image of the storage compartment 110 (1110).
  • the refrigerator 100 may photograph the inside of the storage compartment 110 through the camera 150 and acquire an internal image of the storage compartment 110.
  • the controller 190 may control the camera 150 to photograph the inside of the storage compartment 110 when the closing of the open door 120 is detected.
  • the camera 150 may photograph a plurality of spaces partitioned by the shelf 110c.
  • the first imager 151 of the camera 150 photographs the first storage space 111a of the upper storage room 111
  • the second imager 151 captures the second storage space 111b.
  • the third imager 151 may photograph the third storage space 111c.
  • the controller 190 may acquire an internal image of the storage compartment 110 from the camera 150.
  • the controller 190 may control the camera 150 to photograph the inside of the storage compartment 110 in response to a user input through the touch screen display 130, and the camera 150 may control the storage compartment 110 from the camera 150. An internal image may be acquired.
  • the controller 190 may display an internal image of the storage compartment 110 captured by the camera 150 on the touch screen display 130.
  • the controller 190 may include an internal image 610 of the first storage space 111a, an internal image 620 of the second storage space 111b, and an internal image (of the third storage space 111c). 630 may be displayed on the touch screen display 130, respectively.
  • the refrigerator 100 transmits an internal image of the storage compartment 110 to the server apparatus 200, and the server apparatus 200 receives an internal image of the storage compartment 110 from the refrigerator 100 (1120).
  • the refrigerator 100 may transmit an internal image of the storage room 110 to the server device 200 through a communication network NET.
  • the controller 190 may control the communicator 170 to transmit the internal image of the storage room 110 to the server device 200.
  • the server device 200 may receive an internal image of the storage compartment 110 from the refrigerator 100 through a communication network NET.
  • the processor 240 may acquire an internal image of the storage compartment 110 through the communicator 220.
  • the server device 200 identifies the food included in the internal image of the storage compartment 110 (1130).
  • the server device 200 may identify the food included in the internal image of the storage room 110 using the trained object identification engine 231.
  • the processor 240 may identify the food areas 621, 622, and 623 in which the foods are located in the storage room interior image 620. For example, the processor 240 may display the food regions based on a difference between an internal image 602 of the empty storage compartment 110 and an internal image 620 of the storage compartment 110 in which the food is placed, as shown in FIG. 20. 621, 622, and 623 may be separated from the background region. The processor 240 may identify the first food region 621 in which pineapples are located, the second food region 622 in which canned foods are located, and the third food region 623 in which grapes and apples overlap. have.
  • the processor 240 may classify the food areas 621, 622, and 623. For example, the processor 240 may determine the food areas 621, 622, and 623 as a letter identification area where a letter is detected in the food area, an image identification area where a single food is detected in the food area, and a food area. The food may be classified into an image segmentation area in which a plurality of foods are detected.
  • the processor 240 may classify the first food region 621 in which the pineapple is located as the image identification region, and classify the second food region 622 in which the canned food is located as the character identification region. 3 may be classified as an image segmentation region in which the third food regions 623 overlapping each other are positioned.
  • the processor 240 may identify the pineapple by using an image identification algorithm with respect to the first food region 621, and identify canned food by using a character identification algorithm with respect to the second food region 622. In addition, the processor 240 may separate the grape image and the apple image from the third food region 623, and identify the grape and the apple by using an image identification algorithm for each of the grape image and the apple image. .
  • the processor 240 may include an object identification engine 231 using a neural network.
  • the processor 240 may input luminance values and / or color values of the plurality of pixels included in the image to each of the plurality of input nodes i1 and i2 of the neural network.
  • the processor 240 may apply the weights w1 to w4 to the values of the plurality of input nodes i1 and i2 and output the result to the plurality of hidden nodes h1 and h2.
  • the processor 240 inputs values input to the plurality of hidden nodes h1 and h2 to the sigmoid function, and applies the weights w5-w8 to the output values of the sigmoid function to output the plurality of output nodes.
  • the processor 240 inputs values input to the plurality of output nodes o1 and o2 to the sigmoid function, and the output value of the sigmoid function becomes an output of the neural network.
  • food is allocated to each of the plurality of output nodes o1 and o2
  • an output value of the plurality of output nodes o1 and o2 is an object in which an image is allocated to the plurality of output nodes o1 and o2. It can represent the probability.
  • the processor 240 may identify food based on the outputs of the plurality of output nodes o1 and o2.
  • the processor 240 may collect information (food related information) related to the food identified from the respective food areas 621, 622, and 623.
  • the food-related information may include an image of the food (eg, a location of the food image in the storage compartment image), a name of the food, a category, and a refrigeration (or freezing) storage period.
  • the processor 240 may integrate information related to the food identified from the respective food areas 621, 622, and 623.
  • the server device 200 transmits food related information to the refrigerator 100, and the refrigerator 100 receives food related information from the server device 200 (1140).
  • the server device 200 may transmit food related information to the refrigerator 100 through a communication network NET.
  • the processor 240 may control the communicator 220 to transmit food-related information to the refrigerator 100.
  • the refrigerator 100 may receive food related information from the server device 200 through a communication network NET.
  • the controller 190 may obtain food related information through the communicator 170.
  • the refrigerator 100 displays food related information received from the server device 200 (1150).
  • the controller 190 may display food-related information received from the server device 200 on the touch screen display 130.
  • the controller 190 may further display food-related information 621a, 622a, 623a, and 623b on the internal images 610, 620, and 630 of the storage compartment 110.
  • the controller 190 may include information 621a about pineapples, information 622a about canning, and grapes on the internal images 610, 620, and 630 of the storage compartment 110.
  • Information 623a and information 623b about the apple can be displayed.
  • the controller 190 may display the name of the pineapple, the name of canned food, the name of grapes, and the name of an apple on the internal images 610, 620, and 630 of the storage room 110.
  • the controller 190 may display a list 640 of food information stored in the storage compartment 110. As illustrated in FIG. 22, the controller 190 may display information 641 regarding pineapples, information 642 about canning, information 643 about grapes, and information 644 about apples. have.
  • the controller 190 may display an image of a food (eg, an image of a food region separated from an image of a storage compartment), a name of a food, a category, and a storage date of a refrigerated (or frozen) product on the touch screen display 130. have.
  • the controller 190 may display the image 641a of the pineapple, the name 641b of the pineapple and the storage period 641c of the pineapple included in the internal image of the storage compartment 110. Can be.
  • the server device 200 transmits food related information to the user device 300, and the user device 300 receives food related information from the server device 200 (1160).
  • the user device 300 displays food related information received from the server device 200 (1170).
  • the server device 200 may receive an image of food stored from the refrigerator 100 and identify the food stored in the refrigerator 100 using the object identification engine 231.
  • the refrigerator 100 may display food-related information received from the server device 200.
  • the refrigerator 100 may display information related to food stored in the storage compartment 110 without a user's input.
  • FIG. 23 is a view illustrating another example of a food identification method of the food management system according to one embodiment.
  • FIG. 24 illustrates an example of receiving a user's touch input by the food identification method illustrated in FIG. 23.
  • FIG. 25 illustrates an example of extracting a sample image by the food identification method illustrated in FIG. 23.
  • FIG. 26 illustrates an example of displaying food related information by the food identification method illustrated in FIG. 23.
  • FIG. 27 illustrates another example of displaying food-related information by the food identification method illustrated in FIG. 23.
  • the refrigerator 100 acquires an internal image of the storage compartment 110 (2110).
  • the refrigerator 100 may photograph the inside of the storage compartment 110 through the camera 150 and acquire an internal image of the storage compartment 110.
  • Operation 2110 may be the same as operation 1110 illustrated in FIG. 18.
  • the refrigerator 100 transmits an internal image of the storage compartment 110 to the server apparatus 200, and the server apparatus 200 receives an internal image of the storage compartment 110 from the refrigerator 100 (2120).
  • Operation 2120 may be the same as operation 1120 illustrated in FIG. 18.
  • the server device 200 stores an internal image of the storage room 110 in operation 2130.
  • the processor 240 may store an internal image of the storage compartment 110 received from the refrigerator 100 in the storage 230.
  • the processor 240 may store a predetermined number of internal images.
  • the processor 240 may sort the internal image of the storage compartment 110 according to the order received from the refrigerator 100, and delete the internal image received the longest in response to the new internal image being received.
  • the refrigerator 100 receives a user's touch input from the user (2140).
  • the refrigerator 100 may display an internal image of the storage compartment 110 captured by the camera 150 on the touch screen display 130.
  • the controller 190 displays an internal image 610 of the first storage space 111a, an internal image 620 of the second storage space 111b, and an internal image 630 of the third storage space 111c. To control the touch screen display 130.
  • the user may touch the touch screen display 130 on which the internal image of the storage compartment 110 is displayed.
  • the user may touch the touch screen display 130 at a position corresponding to the image of the food in order to set a storage deadline of the newly received food.
  • the user may touch the inside of the pineapple image or the periphery of the pineapple image of the internal image 620 of the second storage space 111b.
  • the touch screen display 130 may detect touch coordinates of a user's touch input and provide the touch coordinates to the controller 190.
  • the controller 190 may receive touch coordinates of the touch input from the touch screen display 130.
  • the refrigerator 100 receiving the user's touch input transmits a food identification request to the server device 200, and the server device 200 receives a food identification request from the refrigerator 100 (2150).
  • the refrigerator 100 may transmit a food identification request to the server device 200 through a communication network NET.
  • the controller 190 may control the communicator 170 to transmit a food identification request to the server device 200 in response to receiving the user's touch input.
  • the server device 200 may receive a food identification request from the refrigerator 100 through a communication network NET.
  • the processor 240 may receive a food identification request through the communicator 220.
  • the server device 200 identifies the food included in the internal image of the storage compartment 110 (1130).
  • the server device 200 may identify the food included in the internal image of the storage room 110 using the trained object identification engine 231.
  • the processor 240 may extract the plurality of sample images 651 to 657 from the image 620 inside the storage room.
  • the processor 240 may extract sample images 651 to 657 having an arbitrary size (horizontal length, vertical length) at an arbitrary position (x-axis coordinate, y-axis coordinate).
  • the processor 240 selects an x-axis coordinate (x), a y-axis coordinate (y), a horizontal length (h), and a vertical length (v) of the sample image by using a random function, and selects the selected coordinates (x, y) and
  • An image having sizes h and v may be extracted from the storage interior image 620.
  • the processor 240 may include the first sample image 651, the second sample image 652, the third sample image 653, and the n th sample from the interior image 620 of the storage chamber. The image can be extracted.
  • the number of sample images is not limited.
  • the processor 240 may identify foods included in the storage interior image 620 based on the plurality of sample images 651 to 657 using the trained object identification engine 231.
  • the processor 240 may identify foods included in each of the plurality of sample images 651 to 657 using an image identification algorithm using a neural network.
  • an image identification algorithm using a neural network may output a numerical value (for example, probability) indicating which food among foods included in the sample image corresponds to a predetermined food.
  • the processor 240 may select a food having the largest numerical value output from the neural network as the first candidate food, and select a food having the second largest numerical value output from the neural network as the second candidate food.
  • the processor 240 may output pine apple as the first candidate food of the second sample image 652, and output carrots as the second candidate food of the second sample image 652.
  • the processor 240 may output the apple as the first candidate food of the seventh sample image 657 and output the onion as the second candidate food of the seventh sample image 657.
  • the processor 240 may determine the coordinates and sizes of the plurality of sample images 651 to 657, the first candidate food of the plurality of sample images 651 to 657, and the plurality of sample images 651 to 657. Identification data including the second candidate foods may be output.
  • the server device 200 transmits identification data to the refrigerator 100, and the refrigerator 100 receives identification data from the server device 200 (2170).
  • the server device 200 may transmit identification data to the refrigerator 100 through a communication network NET.
  • the processor 240 may control the communicator 220 to transmit identification data to the refrigerator 100.
  • the refrigerator 100 may receive identification data from the server device 200 through a communication network NET.
  • the controller 190 may receive identification data through the communication unit 170.
  • the refrigerator 100 selects a sample image based on the touch coordinates of the user (2180).
  • the refrigerator 100 compares the touch coordinates when the user touches the touch screen display 130 with the center coordinates of the plurality of sample images 651 to 657 and displays a sample image having the center coordinates closest to the user's touch coordinates. You can choose.
  • the controller 190 may obtain touch coordinates of the user's touch input from the touch screen display 130 in operation 2140.
  • the controller 190 may include coordinates and sizes indicating the plurality of sample images 651 to 657, first candidate foods of the plurality of sample images 651 to 657, and a plurality of sample images. Identification data including the second candidate foods 651-657 may be received.
  • the controller 190 may determine the coordinates of the center of the sample images 651 to 657 from the coordinates and the sizes of the sample images 651 to 657. For example, the controller 190 may determine the coordinate x of the center of the sample images 651 to 657 from the sum of the coordinates (x, y) and the half of the size (h, v) of the sample images 651 or 657. + h / 2, y + v / 2) can be determined.
  • the controller 190 may determine the distance between the user's touch input and the center of each of the plurality of sample images 651 to 657. For example, the controller 190 may determine the distance between the touch input of the user and the center of the sample image from the sum of the squares of the differences between the coordinates of the touch input and the center coordinate of the sample image.
  • the controller 190 may identify a sample image having a minimum distance between a center of the plurality of sample images 651 to 657 and a user's touch input.
  • the controller 190 may include a center between the center of the second sample image 652 and the user's touch input P1 among the plurality of sample images 651 to 657. It can be determined that the distance is minimum. The controller 190 may select the second sample image 652 as the image corresponding to the user's touch input P1.
  • the refrigerator 100 displays an identification result of the food selected by the user (2190).
  • the refrigerator 100 may determine a sample image whose center is the minimum distance from the user's touch input. For example, the refrigerator 100 may select the second sample image 652 as an image corresponding to the user's touch input P1.
  • the controller 190 may determine that a sample image having a minimum distance from the user's touch input is an image representing food at a location touched by the user. In addition, the controller 190 may determine that the center of the first candidate food or the second candidate food identified by the user is a food selected by the user based on a sample image having a minimum distance from the user's touch input.
  • the controller 190 may display an identification result of the food selected by the user around the user's touch input.
  • the controller 190 may display food identification information 624 around the user's touch input on the storage interior image 600.
  • the food identification information 624 includes a name area 624a for indicating the name of the food, a name (pineapple) of the first candidate food 624a, and a name (carrot) of the second candidate food 624c. can do.
  • the user may select one of a name (pineapple) of the first candidate food 624a and a name (carrot) of the second candidate food 624c.
  • the name of the food selected by the user may be displayed in the name area 624a.
  • the controller 190 may display “pineapple” in the name area 624a.
  • the controller 190 may collect information about the food based on the name of the food selected by the user. For example, the controller 190 may collect a category of food selected by the user, a storage deadline of the food, and the like. For example, the controller 190 may collect the food category, storage period, etc. of the pineapple.
  • the controller 190 may integrally store a sample image, a name, a category, a storage period, and the like, representing the food selected by the user.
  • the controller 190 may display the list 670 of the food selected by the user on the touch screen display 130. For example, as illustrated in FIG. 27, the controller 190 may display information 671 about pineapple on the touch screen display 130.
  • the controller 190 may include a sample image 671a closest to the user's touch input on the touch screen display 130, a name 671b of the food identified from the second sample image 625, and a second sample image 625.
  • the storage deadline 671c of the identified food item may be displayed.
  • the controller 190 may display the second sample image 652, the “pineapple”, and the storage period of the pineapple on the touch screen display 130.
  • the server device 200 may receive an image of food stored from the refrigerator 100 and identify the food stored in the refrigerator 100 using the object identification engine 231.
  • the refrigerator 100 may display food-related information received from the server device 200.
  • the refrigerator 100 may display information related to food stored in the storage compartment 110 without a user's input.
  • FIG. 28 illustrates another example of a food identification method of a food management system according to one embodiment.
  • FIG. 29 illustrates an example of identifying additional food regions by the food identification method illustrated in FIG. 28.
  • FIG. 30 illustrates an example of displaying additional food related information by the food identification method illustrated in FIG. 28.
  • the refrigerator 100 displays information related to the food stored in the storage compartment 110 at the first time point in operation 1210.
  • the refrigerator 100 may obtain information related to the food stored in the storage compartment 110 at the first time point and display it.
  • the refrigerator 100 may photograph the inside of the storage compartment 110 at a first time point, and acquire an internal image of the storage compartment 110 captured at the first time point.
  • the refrigerator 100 may transmit an internal image of the storage compartment 110 captured at the first time point to the server device 200.
  • the refrigerator 100 acquires a first internal image 710 photographed at a first time point as illustrated in FIG. 29A, and the server device 200 obtains the first internal image 710. Can be sent to.
  • the server device 200 may identify the food from the internal image of the storage room 110 captured at the first time point using the object identification engine 231, and transmit the information related to the identified food to the refrigerator 100.
  • the server device 200 may identify the pineapple, canned food, grapes, and apples from the first internal image 710, and transmit information on the pineapple, canned food, grapes, and apples to the refrigerator 100.
  • the refrigerator 100 may receive information related to food stored in the storage room 110 at a first time point from the server device 200 and display the received food related information.
  • the refrigerator 100 acquires an internal image of the storage compartment 110 at a second time point in operation 1220.
  • the controller 190 may control the camera 150 to photograph the inside of the storage compartment 110 at the second time point, and the camera 150 may photograph a plurality of spaces partitioned by the shelf 110c.
  • the controller 190 may acquire an internal image of the storage compartment 110 captured at the second time point from the camera 150.
  • the refrigerator 100 may acquire the second internal image 720 of the storage compartment 110 photographed at the second time point, as illustrated in FIG. 29B.
  • the refrigerator 100 identifies the additional food region 721 (1230).
  • the controller 190 may identify the additional food area 721 from the difference between the internal image of the storage compartment 110 captured at the first time point and the internal image of the storage compartment 110 captured at the second time point. For example, the controller 190 may identify the additional food region 721 from the difference between the first internal image 710 captured at the first time point and the second internal image 720 captured at the second time point. have.
  • the refrigerator 100 transmits an image of the additional food region 721 to the server device 200, and the server device 200 receives an image of the additional food region 721 from the refrigerator 100 (1240).
  • Operation 1240 may be the same as operation 1120.
  • the server device 200 identifies the food included in the image of the additional food region 721 (1250).
  • the server device 200 may identify the food included in the image of the additional food region 721 using the trained object identification engine 231.
  • Operation 1250 may be the same as operation 1130.
  • the processor 240 may collect information related to the identified food from the additional food region 721.
  • the information about the food may include an image of the food, a name of the food, a category, and a storage deadline for refrigeration (or freezing).
  • the server device 200 transmits the additional food related information to the refrigerator 100, and the refrigerator 100 receives the additional food related information from the server device 200 (1260).
  • Operation 1260 may be the same as operation 1140.
  • the refrigerator 100 displays information related to food stored in the storage compartment 110 at a second time point in operation 1270.
  • the controller 190 may add additional food-related information to the server device 200 to the information related to the food stored in the storage compartment 110 at the first time point, and as a result, the food storage stored in the storage compartment 110 at the second time point. Related information can be obtained.
  • the controller 190 may display the information related to the food stored in the storage room 110 on the touch screen display 130 at the second time point.
  • the controller 190 may include information about a pineapple 730, information about canning 740, information about grapes 750, and information about apples 760.
  • information 770 about grapes identified from the additional food region 721 may be further displayed.
  • controller 190 may identify the same food stored at different times. For example, the controller 190 may identify the pineapple stored at the first time point and the pineapple newly stored at the second time point.
  • the refrigerator 100 newly adds food to the existing food-related information by using a difference between an internal image of the storage compartment 110 previously photographed and an internal image of the newly captured storage compartment 110. You can add relevant information. As a result, the refrigerator 100 may identify the same food stored at different times.
  • 31 is another example of a food identifying method of a food management system according to one embodiment; 32 and 33 illustrate an example of modifying food related information by the food identification method illustrated in FIG. 31.
  • the refrigerator 100 may photograph the inside of the storage compartment 110 and acquire an internal image of the storage compartment 110. In addition, the refrigerator 100 may transmit the captured internal image of the storage compartment 110 to the server device 200.
  • the server device 200 may identify the food from the internal image of the storage room 110 using the object identification engine 231, and transmit the information related to the identified food to the refrigerator 100.
  • the refrigerator 100 may receive information related to food stored in the storage room 110 from the server device 200 and display the received food related information.
  • the refrigerator 100 receives the user's food-related information correction (1320).
  • the user may check the food related information displayed on the touch screen display 130 of the refrigerator 100 and may modify the food related information.
  • the controller 190 may display an information display window 810 displaying information related to the identified food.
  • the information display window 810 may further include a confirmation button 811 for confirming information related to the identified food and a cancel button 812 for correcting information related to the food.
  • the information confirmation screen 810 may display wrong information about the food (incorrectly identifying “apple” as “onion”).
  • the user can touch (press) the cancel button 812 and correct wrong information about the food.
  • the refrigerator 100 may display the food-related information 820 to be modified and display a keyboard 830 for inputting new food-related information, as shown in FIG. 33. have.
  • the user may modify the name of the food from "onion” to "apple” using the keyboard 830.
  • the refrigerator 100 displays the food related information modified by the user (1330).
  • the controller 190 may display the food-related information modified by the user on the touch screen display 130.
  • the refrigerator 100 transmits the food-related information modified by the user to the server device 200, and the server device 200 receives the food-related information modified by the user (1340).
  • the refrigerator 100 may transmit the image indicating the food modified by the user and the food-related information modified by the user to the server device 200 through the communication network NET.
  • the controller 190 may control the communicator 170 to transmit the image of the food and the modified food related information to the server device 200.
  • the server device 200 may receive an image of food and modified food related information from the refrigerator 100 through a communication network NET.
  • the processor 240 may acquire an image of the food and the modified food related information through the communication unit 220.
  • the server device 200 trains the object identification engine 231 again (1350).
  • the processor 240 may store the image of the food received through the communication unit 220 and the modified food related information in the storage 230.
  • the image of the food and the modified food related information may be user modified data 233.
  • the processor 240 may retrain the object identification engine 231 using the user modification data 233. For example, the processor 240 may input an image of the food modified by the user to the object identification engine 231, and compare the output of the object identification engine 231 with the food-related information modified by the user. In addition, the processor 240 may update the object identification engine 231 based on the comparison result.
  • the processor 240 may retrain the object identification engine 231 at various times.
  • the processor 240 may retrain the object identification engine 231 using the user modification data 233 at predetermined intervals.
  • the processor 240 may retrain the object identification engine 231 using the user correction data 233.
  • the processor 240 may retrain the object identification engine 231 using the user modification data 233.
  • the processor 240 may retrain the object identification engine 231 using various data.
  • the processor 240 may retrain the object identification engine 231 using data in which the training data 232 and the user modification data 233 are mixed.
  • the ratio of the training data 232 and the user modification data 233 may be predetermined or set by an administrator or a user.
  • the processor 240 may retrain the object identification engine 231 using only the user modification data 233.
  • the user can modify the information related to the food identified by the server device 200, the server device 200 using the food-related information modified by the user object identification engine 231 Can be retrained.
  • the server device 200 provided separately from the refrigerator 100 identifies food from a food image.
  • the server device 200 is not limited to being separately provided in the refrigerator 100.
  • the server device 200 may be provided integrally with the refrigerator 100.
  • the refrigerator 100 may identify food from the food image.
  • the refrigerator 100 may include the area detection engine 251, the area classification engine 252, and the identification engine 253 illustrated in FIG. 11 described above.
  • 34 is a view illustrating a method of identifying food by a refrigerator according to one embodiment.
  • the refrigerator 100 acquires an internal image of the storage compartment 110 (1410).
  • the refrigerator 100 may photograph the inside of the storage compartment 110 through the camera 150 and acquire an internal image of the storage compartment 110.
  • Operation 1410 may be the same as operation 1110.
  • the refrigerator 100 detects food regions from the internal image of the storage compartment 110 (1420).
  • the controller 190 may identify the food areas where the foods are located in the image inside the storage room. For example, the controller 190 may separate the food regions and the background region based on the difference between the internal image of the empty storage compartment 110 and the internal image of the storage compartment 110 in which the food is placed.
  • the refrigerator 100 classifies the identified food areas (1430).
  • the controller 190 may classify the food regions according to a food identification method or a food identification difficulty. For example, the controller 190 may classify the food areas into a letter identification area where a letter is detected, an image identification area where a single food is detected, and an image segmentation area where a plurality of foods are detected.
  • the refrigerator 100 identifies food in the food area (1440).
  • the controller 190 identifies foods in the classified food area in various ways.
  • the controller 190 may identify food using a character identification algorithm in the character identification area, and identify food using an image identification algorithm in the image identification area.
  • the controller 190 may separate food images included in the image segmentation area, and identify foods using the character identification algorithm or the image identification algorithm on the separated food images.
  • the controller 190 may include an object recognition engine using a neural network.
  • the controller 190 may input luminance values and / or color values of the plurality of pixels included in the image to each of the plurality of input nodes i1 and i2 of the neural network.
  • the controller 190 may apply the weights w1-w4 to the values of the plurality of input nodes i1 and i2 and output the result to the plurality of hidden nodes h1 and h2.
  • the controller 190 inputs values input to the plurality of hidden nodes h1 and h2 to the sigmoid function, and applies the weights w5-w8 to the output values of the sigmoid function to output the plurality of output nodes. Can be output to (o1, o2).
  • the processor 240 inputs values input to the plurality of output nodes o1 and o2 to the sigmoid function, and the output value of the sigmoid function becomes an output of the neural network.
  • food is allocated to each of the plurality of output nodes o1 and o2
  • an output value of the plurality of output nodes o1 and o2 is an object in which an image is allocated to the plurality of output nodes o1 and o2. It can represent the probability.
  • the controller 190 may identify the food based on the outputs of the plurality of output nodes o1 and o2.
  • the refrigerator 100 displays information related to the identified food item (1450).
  • the controller 190 may display the information related to the identified food on the touch screen display 130.
  • the controller 190 may display an image of a food (for example, an image of a food region separated from an image of a storage compartment), a name of a food, a category, and a storage date of a refrigerated (or frozen) food, etc., in the touch screen display 130. Can be marked on.
  • the refrigerated (or frozen) shelf life represents the deadline for the user to safely consume food.
  • the refrigerator 100 may identify the food stored in the storage compartment 110 by itself, and display information related to the food stored in the storage compartment 110. In other words, the refrigerator 100 may display information related to food stored in the storage compartment 110 without a user's input.
  • FIG. 35 illustrates another example of an identification method in which the food management system individually identifies each of the foods.
  • FIG. 36 illustrates an example of manually inputting food related information by the food identification method illustrated in FIG. 35.
  • the refrigerator 100 displays an internal image of the storage compartment 110 (1510).
  • the refrigerator 100 may photograph the inside of the storage compartment 110 through the camera 150 and acquire an internal image of the storage compartment 110.
  • the refrigerator 100 may display on the touch screen display 130 including an internal image of the storage compartment 110 photographed by the camera 150.
  • Operation 1510 may be the same as operation 1110.
  • the refrigerator 100 receives a selection for a food area to be registered from the user (1520).
  • the user may select a food region through the touch screen display 130.
  • the user may select the food region 840 by touching the touch screen display 130 corresponding to the internal image of the storage compartment 110.
  • the controller 190 may display the food region 840 selected by the user to be distinguished from other regions. For example, as shown in FIG. 36, an area other than the food area 840 may be darkly displayed. In addition, the controller 190 may display a food add button 850 for registering a food included in the food area 840 on the touch screen display 130.
  • the refrigerator 100 may register a food included in the food area 840 selected by the user.
  • the refrigerator 100 determines whether the reliability of food identification is greater than the reference value (1530).
  • the reliability of food identification may indicate the reliability of the food identification operation by the refrigerator 100 or the server device 200.
  • Reliability may represent, for example, the ratio of the number of times the identified food is registered without the user's modification to the number of times the overall food identification operation.
  • the reliability may represent a rate at which the refrigerator 100 or the server device 200 succeeds in identifying food.
  • the reference value is a criterion for evaluating the food identification reliability of the refrigerator 100 or the server device 200, and may be set in advance by a designer or post-management by a manager or a user.
  • the refrigerator 100 may calculate the reliability of the food identification based on the result of the previous food identification operation and the correction of the food-related information of the user, and compare the reliability of the food identification with the reference value.
  • the refrigerator 100 automatically registers the food in the selected food region 840 (1540).
  • the refrigerator 100 may identify a food in the food area 840 and automatically register the identified food.
  • the refrigerator 100 transmits an image of the food area 840 selected by the user to the server device 200, and the server device 200 uses the object identification engine 231 to transmit the food area 840.
  • Food can be identified from the image.
  • the refrigerator 100 may receive food information from the server device 200 and register the received food information.
  • the refrigerator 100 may identify the food directly from the image of the food region 840 using the object identification engine 231. In addition, the refrigerator 100 may register the information of the identified food.
  • the refrigerator 100 manually registers the food of the selected food region 840 (1550).
  • the refrigerator 100 may receive food related information of the food area 840 from a user.
  • the refrigerator 100 may display an image of the food region 840 to receive food-related information of the food region 840 and display a keyboard for inputting food-related information.
  • the user may input food-related information such as the name of the food by using the keyboard.
  • the refrigerator 100 may register the information of the input food.
  • the refrigerator 100 displays food related information (1560).
  • the controller 190 may display the registered food related information on the touch screen display 130.
  • the food-related information may include an image of the food (eg, a location of the food image in the storage compartment image), a name of the food, a category, and a storage date of the refrigerated (or frozen) food.
  • the refrigerator 100 may selectively register foods included in the food area selected by the user.
  • the refrigerator 100 may identify the food and register the identified food automatically (using an object identification engine) or manually (according to a user's input) according to the reliability of the food identification.
  • FIG. 37 illustrates another example of an identification method of collectively identifying food items by a food management system.
  • FIG. 38 illustrates an example of modifying food related information by the food identification method illustrated in FIG. 37.
  • FIG. 39 illustrates another example of modifying food related information by the food identification method illustrated in FIG. 37.
  • the refrigerator 100 displays an internal image of the storage compartment 110 in operation 1610.
  • the refrigerator 100 may photograph the inside of the storage compartment 110 through the camera 150 and acquire an internal image of the storage compartment 110.
  • the refrigerator 100 may display on the touch screen display 130 including an internal image of the storage compartment 110 photographed by the camera 150.
  • Operation 1610 may be the same as operation 1110.
  • the refrigerator 100 receives an input for collectively registering foods from the use place (1620).
  • the user may input an input for collectively registering foods.
  • the refrigerator 100 may identify foods included in an internal image of the storage compartment 110.
  • the refrigerator 100 transmits an image of the food region selected by the user to the server apparatus 200, and the server apparatus 200 identifies the food from the image of the food region using the object identification engine 231. can do.
  • the refrigerator 100 may identify the food directly from the image of the food region using the object identification engine 231.
  • the refrigerator 100 displays information related to the identified food (1630).
  • the controller 190 may display the identified food related information on the touch screen display 130.
  • the food-related information may include an image of the food (eg, a location of the food image in the storage compartment image), a name of the food, a category, and a storage date of the refrigerated (or frozen) food.
  • the refrigerator 100 receives modification of food related information of the user (1640).
  • the user may check the food related information displayed on the touch screen display 130 of the refrigerator 100 and may modify the food related information.
  • the refrigerator 100 may display a list including information about pineapples, information about canned food, information about grapes, and information about apples.
  • the refrigerator 100 may display erroneous information about the “apple” (incorrectly identifying “apple” as “onion”).
  • the user can correct wrong information about the "apple”.
  • the user may touch the correction button 860 displayed on the touch screen display 130.
  • the refrigerator 100 may display a food information input screen.
  • the food information input screen may include a keyboard for inputting an image of food to be modified and food related information. The user can change the name of the food from "onion" to "apple” using the keyboard.
  • the refrigerator 100 displays the pineapple information, the canning information, the grape information, and the apple information on the inner image of the storage compartment 110 by overlapping them. can do.
  • the user may touch the correction button 861 displayed on the touch screen display 130.
  • the refrigerator 100 may display a food information input screen.
  • the food information input screen may include a keyboard for inputting an image of food to be modified and food related information. The user can change the name of the food from "onion" to "apple” using the keyboard.
  • the refrigerator 100 displays the food-related information modified by the user (1650).
  • the controller 190 may display the food-related information modified by the user on the touch screen display 130.
  • the refrigerator 100 may collectively register food items stored in the storage compartment 110.
  • the refrigerator 100 may modify food-related information according to a user's input.
  • FIG. 40 is a view illustrating an example of a food management method of a food management system according to one embodiment.
  • FIG. 41 illustrates an example of purchasing food by the food management method illustrated in FIG. 40.
  • the refrigerator 100 stores information related to food stored in the storage compartment 110 (1710).
  • the refrigerator 100 may photograph the inside of the storage compartment 110 and transmit the photographed internal image of the storage compartment 110 to the server device 200.
  • the server device 200 may identify the food from the internal image of the storage room 110 using the object identification engine 231, and transmit the information related to the identified food to the refrigerator 100.
  • the refrigerator 100 may receive information related to food stored in the storage room 110 from the server device 200 and store the received food related information.
  • the food-related information may include an image of the food (eg, a location of the food image in the storage compartment image), a name of the food, a category, and a refrigeration (or freezing) storage period.
  • the refrigerator 100 may photograph the inside of the storage compartment 110, and identify food from the captured interior image of the storage compartment 110.
  • the refrigerator 100 may store information related to the identified food.
  • the refrigerator 100 determines whether the remaining storage period of the food is shorter than the reference period (1720).
  • the controller 190 may calculate a remaining storage time from the date when the food is stored (cold or frozen) and the date the food is stored and the current date from the food-related information, and compare the remaining storage time with the reference time.
  • the reference deadline is a deadline for warning the expiration of the refrigerated (or frozen) shelf life of the food and recommending the ingestion of the food, which may be preset by the designer or post adjusted by the user.
  • the refrigerator 100 determines whether to purchase food with a remaining storage deadline shorter than the reference deadline (1730).
  • the controller 190 may determine whether to purchase food based on a user input.
  • the controller 190 may display a food purchase window 870 for purchasing a food whose residual storage period is shorter than the reference period.
  • the food purchase window 870 displays information related to a food whose remaining storage period is shorter than a reference period, and may include a confirmation button 871 for purchasing a food and a cancel button 872 for canceling a food purchase. .
  • the refrigerator 100 If it is determined that the food is purchased (YES in 1730), the refrigerator 100 requests a food purchase from the online shopping server (1740).
  • the controller 190 may access an online shopping server through a communication network NET.
  • the controller 190 may request the purchase of food from the online shopping server and pay for the food using the payment means set in advance by the user.
  • the remaining storage deadline of the food is not shorter than the reference deadline (NO in 1720) or determined not to purchase the food (NO in 1730), or after the purchase of the food, the refrigerator 100 displays food-related information (1750).
  • the controller 190 may display food related information on the touch screen display 130. For example, when purchasing food, the controller 190 may display information related to the purchased food together.
  • the refrigerator 100 may provide the user with information about the purchase of food based on the food-related information, and purchase the food that is about to expire due to the user's input.
  • FIG. 42 is a view illustrating another example of a food management method of the food management system according to one embodiment.
  • the refrigerator 100 stores information related to food stored in the storage compartment 110 (1810).
  • Operation 1810 may be the same as operation 1710.
  • the refrigerator 100 identifies a user preferred food (1820).
  • the controller 190 may store information related to the stored food when the new food is stored in the storage room 110, and collect information related to the extracted food when the food stored in the storage room 110 is withdrawn.
  • the controller 190 may analyze a purchase pattern for purchasing the same (or similar) food and a consumption pattern for consuming the food based on the information related to the stored food and the information related to the extracted food.
  • the controller 190 may identify the user's preferred food based on the purchase pattern and the consumption pattern of the food.
  • the refrigerator 100 obtains sales information of the user's preferred food (1830).
  • the controller 190 may request sales information of the user's preferred food from the online shopping server through the communication network NET, and receive sales information of the user's preferred food from the online shopping server.
  • the sales information may include price information, discount information, inventory information, and the like of the user's preferred food.
  • the refrigerator 100 determines whether to purchase a user preferred food (1840).
  • the controller 190 may determine whether to purchase food based on a user input. For example, the controller 190 may display the sales information of the user's preferred food on the touch screen display 130 and receive a user input regarding the purchase of the user's preferred food through the touch screen display 130.
  • the refrigerator 100 If it is determined that the food is purchased (Yes of 1840), the refrigerator 100 requests a food purchase from the online shopping server (1850).
  • Operation 1850 may be the same as operation 1740.
  • the refrigerator 100 displays food-related information (1860).
  • the controller 190 may display food related information on the touch screen display 130. For example, when purchasing food, the controller 190 may display information related to the purchased food together.
  • the refrigerator 100 may provide the user with information regarding the purchase of the food based on the storage and withdrawal record of the food, and purchase the user's preferred food according to the user input.
  • FIG. 43 is a view illustrating a recipe providing method of a food management system according to one embodiment.
  • 44 and 45 illustrate an example of providing a recipe by the method of providing a recipe illustrated in FIG. 43.
  • FIG. 46 illustrates another example of providing a recipe by the recipe providing method of FIG. 43.
  • the refrigerator 100 displays information related to the food stored in the storage compartment 110 (1910).
  • Operation 1910 may be the same as operation 1310.
  • the refrigerator 100 receives a recipe request from a user (1920).
  • the user may select at least some of the food-related information displayed on the touch screen display 130 and request a recipe including a food related to the selected information.
  • the controller 190 may display a check box 910 for selecting food-related information together with food-related information.
  • the controller 190 may display a recipe request window 920 for requesting a recipe, as shown in FIG. 45, and the recipe request window 920 may include a request button 921 and a cancel button 922. It may include.
  • the refrigerator 100 may receive a recipe request including the checked food.
  • the controller 190 may display an internal image of the storage room 110 on the touch screen display 130, and receive a recipe request from the user through the touch screen display 130. As illustrated in FIG. 46, when the user touches the food image 911 for a long time, the controller 190 may display a recipe request popup 912. When the user touches the recipe request popup 912, the controller 190 may receive a recipe request including a food of the food image 911.
  • the refrigerator 100 transmits the food related information and the recipe request selected by the user to the server device 200, and the server device 200 receives the food related information and the recipe request from the refrigerator 100 in 1930.
  • the server device 200 searches for a recipe including a food selected by a user (1940).
  • the processor 240 may search for a recipe stored in the storage 230 or request a recipe from other servers through a communication network NET, and obtain a recipe including a food selected by a user.
  • the server device 200 transmits information about the recipe to the refrigerator 100, and the refrigerator 100 receives information about the recipe from the server device 200 (1950).
  • the refrigerator 100 displays a recipe received from the server device 200 (1960).
  • the controller 190 may display a recipe including the food selected by the user on the touch screen display 130.
  • the refrigerator 100 may provide a recipe including a food selected by a user.
  • 47 is a view illustrating a recipe providing method of a food management system according to one embodiment.
  • 48 and 49 illustrate an example of providing a recipe by the method of providing a recipe illustrated in FIG. 47.
  • the refrigerator 100 obtains a cooking image 930 (2010).
  • the refrigerator 100 may obtain a cooking image 930 from a user or by itself.
  • the controller 190 may access the communication network NET through the communication unit 170 in response to a user input, and receive the cooking image 930 from another device connected to the communication network NET.
  • the controller 190 may photograph the inside of the storage compartment 110 using the camera 150, and acquire a cooking image 930 of a dish stored in the storage compartment 110.
  • the refrigerator 100 receives a recipe request from a user in 2020.
  • the user may request a recipe for cooking a dish included in the cooking image 930.
  • the controller 190 may display a cooking image 930 and a recipe request window 940 for requesting a recipe.
  • the recipe request window 940 may include a request button 941 and a cancel button 942.
  • the refrigerator 100 may receive a recipe request of the cooking image 930.
  • the refrigerator 100 transmits the cooking image 930 and the recipe request to the server device 200, and the server device 200 receives the cooking image 930 and the recipe request from the refrigerator 100 in operation 2030.
  • the server device 200 identifies a dish from the cooking image 930 (2040).
  • the processor 240 may identify a dish included in the cooking image 930 using the trained object identification engine 231.
  • the processor 240 may include an object identification engine 231 using a neural network.
  • the processor 240 may input luminance values and / or color values of the plurality of pixels included in the cooking image 930 to the neural network.
  • the processor 240 may identify the cooking of the cooking image 930 based on the output of the neural network.
  • the server device 200 retrieves a recipe for the identified dish (2050).
  • the processor 240 may search for recipes stored in the storage 230 or request recipes from other servers through a network NET, and obtain recipes of the identified dishes.
  • the server device 200 transmits information about the recipe to the refrigerator 100, and the refrigerator 100 receives information about the recipe from the server device 200 (2060).
  • the refrigerator 100 displays the recipe received from the server device 200 (2070).
  • the controller 190 may display a recipe including the food selected by the user on the touch screen display 130.
  • the refrigerator 100 displays insufficient food (2080).
  • the controller 190 compares a list of food items stored in the storage room 110 with a list of ingredients included in a recipe received from the server device 200, and identifies foods not stored in the storage room 110 among the ingredients of the recipe. Can be.
  • the controller 190 may display the food lacking in the recipe material on the touch screen display 130.
  • the controller 190 may display a food lacking in the pasta recipe.
  • the controller 190 may display information 950 related to potatoes, information 960 related to pasta noodles, information 970 related to onions, and information 980 related to tomato paste. 130).
  • the controller 190 may further display a purchase button 990 for purchasing the insufficient food on the touch screen display 130.
  • the controller 190 may request the purchase of food from the online shopping server and pay for the food using the payment means set in advance by the user.
  • the server device 200 may identify a dish from the cooking image 930 received from the refrigerator 100 and search for a recipe of the dish.
  • the refrigerator 100 may receive a recipe for cooking from the server device 200, and may purchase food that is insufficient in the ingredients included in the recipe.
  • the display apparatus 100 may control the local scattering rate of the electro-optical layer 231 to perform local dimming.
  • the disclosed embodiments may be implemented in the form of a recording medium for storing instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may generate a program module to perform the operations of the disclosed embodiments.
  • the recording medium may be implemented as a computer-readable recording medium.
  • Computer-readable recording media include all kinds of recording media having stored thereon instructions which can be read by a computer.
  • ROM read only memory
  • RAM random access memory
  • magnetic tape a magnetic tape
  • magnetic disk a magnetic disk
  • flash memory an optical data storage device, and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Chemical & Material Sciences (AREA)
  • Combustion & Propulsion (AREA)
  • Mechanical Engineering (AREA)
  • Thermal Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Cold Air Circulating Systems And Constructional Details In Refrigerators (AREA)

Abstract

La présente invention concerne un système de gestion des aliments qui comprend : un dispositif serveur ; et un réfrigérateur comprenant une chambre de stockage et pour transmettre une image de la chambre de stockage au dispositif serveur. Le dispositif serveur peut comprendre : une unité de communication ; et une unité de traitement pour identifier des aliments à partir de l'image reçue en provenance du réfrigérateur au moyen de l'unité de communication, et transmettre des informations concernant l'aliment identifié au réfrigérateur au moyen de l'unité de communication. L'unité de traitement peut identifier des aliments au moyen de différents processus à partir de différentes images.
PCT/KR2019/002851 2018-03-13 2019-03-12 Réfrigérateur, et système et son procédé de commande Ceased WO2019177343A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US16/979,586 US11425338B2 (en) 2018-03-13 2019-03-12 Refrigerator, and system and method for controlling same
EP19766651.4A EP3745304B1 (fr) 2018-03-13 2019-03-12 Réfrigérateur, et système et son procédé de commande
CN201980025477.6A CN111971684A (zh) 2018-03-13 2019-03-12 冰箱、其控制方法以及系统

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
KR20180028944 2018-03-13
KR10-2018-0028944 2018-03-13
KR10-2019-0023124 2019-02-27
KR1020190023124A KR102217024B1 (ko) 2018-03-13 2019-02-27 냉장고, 그 제어 방법 및 시스템

Publications (1)

Publication Number Publication Date
WO2019177343A1 true WO2019177343A1 (fr) 2019-09-19

Family

ID=67906753

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2019/002851 Ceased WO2019177343A1 (fr) 2018-03-13 2019-03-12 Réfrigérateur, et système et son procédé de commande

Country Status (1)

Country Link
WO (1) WO2019177343A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112535408A (zh) * 2019-09-23 2021-03-23 佛山市顺德区美的电热电器制造有限公司 辅助烹饪方法、装置和计算机存储介质
CN112782994A (zh) * 2020-12-31 2021-05-11 重庆房慧科技有限公司 基于用户数据分析的智能家居控制系统
CN113465265A (zh) * 2020-04-27 2021-10-01 海信集团有限公司 一种智能冰箱及食材检测方法
CN116263574A (zh) * 2021-12-15 2023-06-16 青岛聚看云科技有限公司 一种智能冰箱、服务器及界面显示方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014206290A (ja) * 2013-04-10 2014-10-30 株式会社東芝 冷蔵庫
KR20140127590A (ko) * 2013-04-25 2014-11-04 엘지전자 주식회사 냉장고 및 그 제어방법
JP2015169412A (ja) * 2014-03-10 2015-09-28 三菱電機株式会社 冷蔵庫及び冷蔵庫を備えたネットワークシステム
KR101756620B1 (ko) * 2017-02-03 2017-07-12 구영모 냉장고의 문자판독 인공지능에 의한 보관식품 유통기한 알림장치와 그 방법
KR20180025041A (ko) * 2016-08-31 2018-03-08 엘지전자 주식회사 냉장고 및 냉장고의 제어 방법

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014206290A (ja) * 2013-04-10 2014-10-30 株式会社東芝 冷蔵庫
KR20140127590A (ko) * 2013-04-25 2014-11-04 엘지전자 주식회사 냉장고 및 그 제어방법
JP2015169412A (ja) * 2014-03-10 2015-09-28 三菱電機株式会社 冷蔵庫及び冷蔵庫を備えたネットワークシステム
KR20180025041A (ko) * 2016-08-31 2018-03-08 엘지전자 주식회사 냉장고 및 냉장고의 제어 방법
KR101756620B1 (ko) * 2017-02-03 2017-07-12 구영모 냉장고의 문자판독 인공지능에 의한 보관식품 유통기한 알림장치와 그 방법

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112535408A (zh) * 2019-09-23 2021-03-23 佛山市顺德区美的电热电器制造有限公司 辅助烹饪方法、装置和计算机存储介质
CN113465265A (zh) * 2020-04-27 2021-10-01 海信集团有限公司 一种智能冰箱及食材检测方法
CN113465265B (zh) * 2020-04-27 2022-04-01 海信集团有限公司 一种智能冰箱及食材检测方法
CN112782994A (zh) * 2020-12-31 2021-05-11 重庆房慧科技有限公司 基于用户数据分析的智能家居控制系统
CN116263574A (zh) * 2021-12-15 2023-06-16 青岛聚看云科技有限公司 一种智能冰箱、服务器及界面显示方法

Similar Documents

Publication Publication Date Title
WO2020096235A2 (fr) Système de gestion des aliments, appareil serveur et réfrigérateur
WO2019177343A1 (fr) Réfrigérateur, et système et son procédé de commande
EP3161398A1 (fr) Réfrigérateur et son procédé de commande
WO2016036015A1 (fr) Réfrigérateur et son procédé de commande
WO2021137345A1 (fr) Réfrigérateur à intelligence artificielle et son procédé de fonctionnement
WO2017111234A1 (fr) Procèdè pour la commande d'un objet par un dispositif èlectronique et dispositif èlectronique
WO2018088794A2 (fr) Procédé de correction d'image au moyen d'un dispositif et dispositif associé
WO2016017945A1 (fr) Dispositif mobile et son procédé d'appariement à un dispositif électronique
WO2015133699A1 (fr) Appareil de reconnaissance d'objet, et support d'enregistrement sur lequel un procédé un et programme informatique pour celui-ci sont enregistrés
WO2019135621A1 (fr) Dispositif de lecture vidéo et son procédé de commande
WO2011105768A2 (fr) Réfrigérateur comprenant un terminal et procédé de commande de celui-ci
WO2016208984A1 (fr) Terminal mobile susceptible de réaliser une commande à distance d'une pluralité de dispositifs
WO2021132851A1 (fr) Dispositif électronique, système de soins du cuir chevelu et son procédé de commande
WO2016013774A1 (fr) Appareil d'achat de biens et système d'achat de biens qui en est doté
WO2021125453A1 (fr) Réfrigérateur à intelligence artificielle
WO2017084337A1 (fr) Procédé, appareil et système de vérification d'identité
WO2018012805A1 (fr) Équipement utilisateur, appareil de cuisson et système de cuisson
WO2017018602A1 (fr) Terminal mobile et procédé de commande correspondant
WO2020017827A1 (fr) Dispositif électronique et procédé de commande pour dispositif électronique
WO2019231235A1 (fr) Dispositif de traitement de signal et dispositif d'affichage d'image comprenant celui-ci
EP3448178A1 (fr) Équipement utilisateur, appareil de cuisson et système de cuisson
WO2024043624A1 (fr) Dispositif de cuisson automatique de viande et son procédé de commande
WO2021206385A1 (fr) Appareil électronique et procédé de commande associé
WO2019132553A1 (fr) Système d'émission de son et procédé de traitement vocal
WO2020213941A1 (fr) Système et procédé permettant de fournir du contenu dans un ordiphone

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19766651

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2019766651

Country of ref document: EP

Effective date: 20200828

NENP Non-entry into the national phase

Ref country code: DE

WWG Wipo information: grant in national office

Ref document number: 2019766651

Country of ref document: EP