EP4088223A1 - Procédé de génération d'une pluralité d'images annotées - Google Patents

Procédé de génération d'une pluralité d'images annotées

Info

Publication number
EP4088223A1
EP4088223A1 EP20811285.4A EP20811285A EP4088223A1 EP 4088223 A1 EP4088223 A1 EP 4088223A1 EP 20811285 A EP20811285 A EP 20811285A EP 4088223 A1 EP4088223 A1 EP 4088223A1
Authority
EP
European Patent Office
Prior art keywords
images
information
pixel
field
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP20811285.4A
Other languages
German (de)
English (en)
Inventor
Markus Hoeferlin
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Robert Bosch GmbH
Original Assignee
Robert Bosch GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Robert Bosch GmbH filed Critical Robert Bosch GmbH
Publication of EP4088223A1 publication Critical patent/EP4088223A1/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/35—Categorising the entire scene, e.g. birthday party or wedding scene
    • G06V20/38—Outdoor scenes
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2155—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/40—Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G06V20/188—Vegetation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/60—Type of objects
    • G06V20/68—Food, e.g. fruit or vegetables

Definitions

  • the present invention relates to a method for generating a plurality of annotated images, and in particular a method for generating a plurality of pixel-by-pixel annotated images of a field on which plants are growing.
  • class is to be understood in the following as a class of a classification and not as a class in the sense of biosystematics.
  • supervised learning methods are predominantly used, such as in "Plant classifications system for crop / weed discrimination without segmentation”; Applications of Computer Vision (WACV), 2014 IEEE Winter Conference, 2014, 1142-1149; "Support Vector Machines for crop / weeds Identification in maize fields”; Expert Systems with Applications, 2012, 39, 11149-11155; and "Evaluation of Features for Leaf Classification in Challenging Conditions”; Applications of Computer Vision (WACV), 2015 IEEE Winter Conference, 2015, 797-804; disclosed.
  • annotation information For the monitored training of the classifier, large amounts of learning or training data are required in the form of images that are annotated or classified with annotation information.
  • This annotation information can have different information.
  • the annotation information preferably includes a classification that shows the pixels in the images that a plant to be classified, ie useful plant or weed, shows correct class too.
  • the annotation information can also have further information, such as row information.
  • the annotation information is also referred to as the ground truth.
  • FIG. 5 shows a tool 50 for annotating images at the pixel level with the annotation information, the annotation information in particular specifying classification information of a plant.
  • the annotation information can be, for example, species, genus or simply a distinction between useful plants and weeds.
  • Each captured image must be annotated separately with annotation information, which is shown by a controller 52.
  • the plants shown are annotated with annotation information that distinguishes between two classes, useful plant (“bed”) 54 and weeds (“other”) 56.
  • semi-automatic annotation support such as following the annotations over several images or frames of a video to be annotated, semi-automatic segmentation of the plants, etc., the effort is very high due to the large number of images to be annotated.
  • the effort for annotating images also increases sharply in crops with smaller plant spacing due to obscurations.
  • 1A shows a picture of the field on which arugula grows, in the visible area
  • 1B shows an image in the infrared range of a field on which arugula is growing
  • 3 shows a stitched picture composed of a plurality of pictures
  • 4A shows a tool for annotating the deleted image
  • a field can be understood to be a delimited area of land for the cultivation of useful plants or a part of such a field.
  • a useful plant is understood to be an agriculturally used plant which itself or its fruit is used, e.g. as food, feed or as an energy plant.
  • the seeds and consequently the plants are primarily arranged in rows, it being possible for objects to be present between the rows and between the individual plants within a row. However, the objects are undesirable because they reduce the yield of the plants or represent a disruptive influence during cultivation and / or harvesting.
  • An object can be understood to mean any plant that is different from the useful plant, or any object. Objects can in particular be weeds, woods and stones.
  • FIG. 1A shows a picture of a field on which arugula is grown in rows, in the visible area.
  • the leaves of each Some of the rocket plants overlap strongly.
  • FIG. 1B the plants can be very robustly segmented from the subsurface by an additionally recorded image of the field in the infrared range and by using various segmentation methods (eg NDVI index, ExG index, ExG - ExR index, etc.)
  • segmentation methods eg NDVI index, ExG index, ExG - ExR index, etc.
  • a pixel can be annotated with further information.
  • annotation information the information with which a pixel is annotated.
  • the method 100 according to the invention is intended to reduce this effort when annotating the multiplicity of images with the annotation information pixel by pixel.
  • the individual steps of the method are shown in the flow chart in FIG. 2 and are described in detail.
  • the multiplicity of images of a surface of the field is acquired by an image acquisition means.
  • the image capturing means is a camera, such as a CCD camera, a CMOS camera, etc., which captures an image in the visible range (see e.g. Fig. 1A) and provides it as RGB values or as values in another color space.
  • the image acquisition means can, however, also be a camera which acquires an image in the infrared range (see e.g. Fig. 1B). An image in the infrared range is particularly suitable for capturing plants, as the reflection of the plants is significantly increased in this frequency range.
  • the image acquisition means can also be, for example, a mono, RGB, multispectral, hyperspectral camera.
  • other data can be acquired using sensors such as 3D sensors, etc. It is possible for several image acquisition means to be present on the vehicle and for several images to be acquired essentially synchronously by the different image acquisition means and data from different sensors.
  • the field on which the plants and objects are present is detected by the image capturing means during a drive over with a vehicle on which the image capturing means is attached.
  • the image acquisition means can be attached to a vehicle specially provided for this purpose. However, the image acquisition means can also be attached to an agricultural vehicle, such as a tractor, a trailer, etc., or an aircraft, such as a drone, for example.
  • the image capturing means is attached to the vehicle in such a way that an image sensor of the image capturing means is in the Is substantially parallel to a surface of the field.
  • the image sensor of the image acquisition means can, however, also be inclined to the surface of the field.
  • the vehicle to which the image capturing means is attached drives or flies the field and the image capturing means captures the images at a predetermined time interval.
  • the images are preferably captured in such a way that they overlap. For this reason, the image capturing means captures several images per second during the passage, as a result of which the images strongly overlap at a low passage speed. In this way, images of a single plant are recorded from several perspectives, so that consequently a large number of images with different views of the same plant are available as learning data. In this way, a recognition accuracy of the classifier is improved.
  • the multitude of images can also be recorded as video.
  • the vehicle can also drive off / fly off the field autonomously or remotely.
  • the images are then stored in a memory and are then available for further processing.
  • the images can be transferred to a separate computing unit that is not attached to the vehicle.
  • the images can also be further processed in a computing unit that is mounted on the vehicle.
  • the large number of images should preferably show all growth phases of the useful plant and, if possible, all occurring weeds with all possible biological growth morphologies, since different circumstances such as water, heat, soil, nutrients, wind, etc., the growth and / or the appearance of the useful plants and the Can affect weeds.
  • step S104 is carried out essentially synchronously with step S102.
  • position information is acquired using position detection means.
  • the position detection means detects the position information using high-precision GPS, but can alternatively also detect it using, for example, odometry, visual odometry, and other tracking methods.
  • the position information is given as world coordinates, but can also be given as field coordinates, longitude + latitude, etc., for example.
  • the position information is then correlated with the image recorded in step S102, so that the position in the field at which the image is recorded can be precisely determined.
  • the position information obtained is assigned to a point in the center of the image.
  • the position information can also be assigned to another point in the image, e.g. a corner point.
  • distances between the mounting position of the image capturing means and the mounting position of the position capturing means on the vehicle must be taken into account when correlating the position information with the captured image.
  • the spatial extent of the image on the field in an X and a Y direction can then be determined using an image angle of the image capturing means and the distance of the image capturing means to the floor surface. If the image sensor of the image acquisition means is inclined to the surface of the field, this inclination must also be taken into account when calculating the spatial extent of the image on the field.
  • position information and consequently a position on the field can also be assigned to each of the pixels of the image.
  • This procedure can also be applied to images that are acquired by another image acquisition means and to data that are acquired by different sensors.
  • step S106 the images that are captured during the same passage and for which the position information was obtained in S104 are combined to form a stitched image using the position information in order to display a coordinate system that is more global than the pixel coordinate system on the image level Has resolution to create.
  • a position in the field that is recorded due to the movement of the vehicle and the rapid repetition rate when recording the images in different images from different perspectives has the same position information in all images.
  • a stitched image as obtained after performing step S106 is shown in FIG. 3.
  • the stitched picture in Fig. 3 was under Using high-precision GPS put together from several hundred images.
  • step S108 the stitched image previously generated in S106 is annotated pixel by pixel with the annotation information, the amount of pixels to be annotated due to the overlap of the images and the representation as a stitched image compared to the case in which all images are to be annotated individually pixel by pixel, is significantly reduced.
  • 4A shows a tool with which the panoramic image is annotated.
  • the useful plants are identified by a marking 42.
  • the pixels belonging to the useful plant around those with marking 42 can be automatically recognized and annotated by the tool and classification information can be added for the individual pixels.
  • the panoramic image can be annotated with row information, i.e. information about the course of a row of plants 44.
  • the annotation information of the pixels of a row of plants 44 thus has the row information in addition to the classification information.
  • the annotation information is again transferred to the individual images using the position information, which were captured in step S102 and from which the stitched image was combined in step S106.
  • the annotation information can also be transferred to images that are only partially shown in the stitched image, since the pixels of the images only partially shown in the stitched image also have the position information.
  • the amount of annotated image material can be increased compared to the representation as a stitched image.
  • the described transmission of the annotated annotation information is not limited to the plurality of images captured by the same image capturing means during the same passage, but can be to the various other images to be annotated and Data collected on the same field can be applied.
  • the prerequisite for this, however, is that these images and data contain the position information and that the images show the same plants during the same cultivation period.
  • the annotation information can also be transferred to the multitude of images showing the same plants (or the same annotation object) at a different point in time with varying perspectives, plant growth stages, lighting conditions, weather conditions,
  • the annotation information can also be transferred to a large number of images that are recorded by another image recording device during the same passage.
  • the annotation information can, however, also be transferred to a large number of images that are recorded by the other image recording means during a passage at a different point in time.
  • the prerequisite for transmitting the annotation information is again that the data to which the annotation information is to be transmitted has the position information.
  • This procedure can also be applied to other data collected by sensors. As a result, the effort for generating a multiplicity of pixel-wise annotated images of a different type and of further data is also reduced.
  • Another advantage of transmitting the annotation information using the position information is that the resolution and the angle of view of the other image capture means do not have to be taken into account.
  • the present invention uses one or more information sources for annotating images with annotation information pixel by pixel in order to have to annotate each plant only once manually in a larger context.
  • the images of the same plant or a part of it, which are not shown in the larger context can then be automatically annotated at the pixel level.
  • a large number of images that have annotation information at the pixel level can be obtained with very little effort.
  • This large number of images, which have the annotation information at the pixel level can then be used for a classifier, such as. For example, to train a neural network, etc., which is to be used for the classification of plants.
  • the method according to the invention is not limited to images of a field on which plants are growing.
  • the method can be used for various applications in which a large number of images that overlap one another are to be annotated with information on a pixel-by-pixel basis.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Image Processing (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

L'invention concerne un procédé (100) pour générer une pluralité d'images annotées pixel par pixel d'un champ dans lequel poussent des plantes, qui comprend les étapes suivantes : le capture (S102) de la pluralité d'images du champ à l'aide d'un moyen de capture d'images, les images se chevauchant mutuellement ; la demande sensiblement simultanée (S104) des informations de position concernant la position à laquelle chacune des images a été capturée dans le champ ; l'agencement (S106) de la pluralité d'images dans un contexte plus global en tant qu'image assemblée, à l'aide des informations de position ; l'annotation (S108) des pixels de l'image assemblée avec des informations d'annotation contenant au moins des informations de classification qui indiquent une classe de la plante ; et le transfert (S110) des informations d'annotation aux pixels de la pluralité d'images, à l'aide des informations de position.
EP20811285.4A 2019-11-25 2020-11-20 Procédé de génération d'une pluralité d'images annotées Withdrawn EP4088223A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102019218189.9A DE102019218189A1 (de) 2019-11-25 2019-11-25 Verfahren zum Generieren einer Vielzahl von annotierten Bildern
PCT/EP2020/082855 WO2021105019A1 (fr) 2019-11-25 2020-11-20 Procédé de génération d'une pluralité d'images annotées

Publications (1)

Publication Number Publication Date
EP4088223A1 true EP4088223A1 (fr) 2022-11-16

Family

ID=73543257

Family Applications (1)

Application Number Title Priority Date Filing Date
EP20811285.4A Withdrawn EP4088223A1 (fr) 2019-11-25 2020-11-20 Procédé de génération d'une pluralité d'images annotées

Country Status (3)

Country Link
EP (1) EP4088223A1 (fr)
DE (1) DE102019218189A1 (fr)
WO (1) WO2021105019A1 (fr)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113538313B (zh) * 2021-07-22 2022-03-25 深圳大学 一种息肉分割方法、装置、计算机设备及存储介质
DE102021207983A1 (de) 2021-07-23 2023-01-26 Robert Bosch Gesellschaft mit beschränkter Haftung Unkrauterkennungsvorrichtung, Verfahren zu Unkrauterkennung, Computerprogramm und Speichermedium
DE102022206224A1 (de) 2022-06-22 2023-12-28 Robert Bosch Gesellschaft mit beschränkter Haftung Verfahren zum automatischen Labeln von Eingangsbildern, Trainingsbilder, Verfahren zum Trainieren oder Nachtrainieren eines Unkrautanalysators, Analysevorrichtung, Computerprogramm sowie maschinenlesbares Speichermedium

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102009023896B4 (de) * 2009-06-04 2015-06-18 Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. Vorrichtung und Verfahren zum Erfassen einer Pflanze

Also Published As

Publication number Publication date
DE102019218189A1 (de) 2021-05-27
WO2021105019A1 (fr) 2021-06-03

Similar Documents

Publication Publication Date Title
EP3866593B1 (fr) Procédé d'épandage d'un produit de pulvérisation sur un champ
EP3782467B1 (fr) Procédé d'identification des mauvaises herbes dans un rang défini de plantes d'une surface agricole
WO2019081349A1 (fr) Production de cartes numériques de traitement
EP4262378B1 (fr) Procédé d'application d'une pulvérisation sur terrain agricole
DE102015221085A1 (de) Verfahren und Informationssystem zum Erkennen zumindest einer auf einem Feld angepflanzten Pflanze
EP4064818B1 (fr) Procédé de traitement de plantes dans un champ
EP4262381B1 (fr) Procédé pour la classification de plantes à des fins agricoles
WO2021105019A1 (fr) Procédé de génération d'une pluralité d'images annotées
WO2021105006A1 (fr) Procédé d'estimation d'une trajectoire de rangées de plantes
US11393193B1 (en) Zone management generation from point samples
WO2021105017A1 (fr) Procédé de traitement de plantes dans un champ
DE102019218192A1 (de) Verfahren zum Bearbeiten von Pflanzen auf einem Feld
DE102017217258A1 (de) Verfahren zum Klassifizieren von Pflanzen
EP4262379B1 (fr) Procédé et dispositif d'application d'un liquide de pulvérisation sur des terres agricoles
EP4245135A1 (fr) Mise en oeuvre et documentation d'une application de produits phytosanitaires
DE102017204650A1 (de) Verfahren zum Klassifizieren von Pflanzen
DE102019209526A1 (de) Überwachen einer Anbaufläche
WO2020039045A1 (fr) Système de support pourvu d'un support et d'un dispositif mobile pour le traitement du sol et/ou pour la manipulation de la flore et de la faune et procédé associé
DE102020215877A1 (de) Verfahren zum Klassifizieren von Pflanzen für landwirtschaftliche Zwecke
Pflanz et al. Automatisierte Unkrauterkennung auf dem Acker–Möglichkeiten und Grenzen
DE102016203850B4 (de) Verfahren zum Klassifizieren von Pflanzen
DE102019201977A1 (de) Verfahren zum Ermitteln eines Zustandes einer Feldfrucht-Bepflanzung und landwirtschaftliches Fahrzeug
DE102019201984A1 (de) Steuergerät zum Einsatz in der Landwirtschaft
DE102023211134A1 (de) Verfahren zum Ansteuern einer Spritzvorrichtung einer Feldspritze
DE102022212161A1 (de) Verfahren zum Ermitteln einer Breitenkennzahl von Pflanzen einer Pflanzenreihe einer landwirtschaftlichen Fläche

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20220929

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20250603