WO2020182564A1 - Système d'aide au pilotage basé sur la vision pour des véhicules terrestres - Google Patents

Système d'aide au pilotage basé sur la vision pour des véhicules terrestres Download PDF

Info

Publication number
WO2020182564A1
WO2020182564A1 PCT/EP2020/055651 EP2020055651W WO2020182564A1 WO 2020182564 A1 WO2020182564 A1 WO 2020182564A1 EP 2020055651 W EP2020055651 W EP 2020055651W WO 2020182564 A1 WO2020182564 A1 WO 2020182564A1
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
processor unit
track line
surroundings
detection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2020/055651
Other languages
German (de)
English (en)
Inventor
Benedict Strohmaier
Costanza CASCELLI
Tommy Ikonen
Jan-Frederik Kuhn
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ZF Friedrichshafen AG
Original Assignee
ZF Friedrichshafen AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ZF Friedrichshafen AG filed Critical ZF Friedrichshafen AG
Publication of WO2020182564A1 publication Critical patent/WO2020182564A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/001—Planning or execution of driving tasks
    • A—HUMAN NECESSITIES
    • A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01B—SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL
    • A01B69/00—Steering of agricultural machines or implements; Guiding agricultural machines or implements on a desired track
    • A01B69/001—Steering by means of optical assistance, e.g. television cameras
    • A—HUMAN NECESSITIES
    • A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
    • A01B—SOIL WORKING IN AGRICULTURE OR FORESTRY; PARTS, DETAILS, OR ACCESSORIES OF AGRICULTURAL MACHINES OR IMPLEMENTS, IN GENERAL
    • A01B69/00—Steering of agricultural machines or implements; Guiding agricultural machines or implements on a desired track
    • A01B69/007—Steering or guiding of agricultural vehicles, e.g. steering of the tractor to keep the plough in the furrow
    • A01B69/008—Steering or guiding of agricultural vehicles, e.g. steering of the tractor to keep the plough in the furrow automatic
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • B60W30/08—Active safety systems predicting or avoiding probable or impending collision or attempting to minimise its consequences
    • B60W30/09—Taking automatic action to avoid collision, e.g. braking and steering
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
    • B60W30/10—Path keeping
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B60W2050/143—Alarm means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2300/00—Indexing codes relating to the type of vehicle
    • B60W2300/15—Agricultural vehicles
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks

Definitions

  • the invention relates to a processor unit, a driver assistance system, a method and a computer program product for the autonomous operation of a vehicle, in particular an agricultural utility vehicle.
  • driver assistance systems for agricultural vehicles or Maschi nen (agricultural technology) known. These driver assistance systems are intended to relieve the driver in particular as a fully automated or partially automated steering aid.
  • the steering aid via GPS control has established itself in agricultural engineering.
  • the driver assistance systems are sometimes very complex and, depending on the correction signal, enable accuracies of up to 2 cm. Problems with GPS are the partially non-existent signal coverage worldwide (e.g. also in mountainous regions) and the high acquisition and license costs, depending on the system.
  • driver assistance systems for agricultural vehicles that are not based on GPS provide optical systems that automate certain applications in agriculture.
  • Examples are driver assistance systems based on cameras, lidar sensors or, more generally, laser sensors.
  • driver assistance systems are complex and expensive (in particular when lidar sensors are used) and / or can only carry out or support a simple application.
  • these systems typically only allow a swath to be recognized in the center in front of the vehicle and not to the side of the vehicle.
  • a “swath” can be understood as an agricultural crop that has been mown and lined up in a row, e.g. Grass, grain, etc.
  • such systems are not able to process several applications (e.g. mowing, swathing, loading or baling) and thus do not offer a real alternative to GPS systems.
  • a driver of an agricultural utility vehicle typically has to concentrate on one operation for a whole day.
  • the driver in addition to controlling the vehicle, the driver usually has to simultaneously monitor the function of attachments in the vehicle. This can cause it to Operating errors mean that the machines are not used optimally efficiently and that work performance and quality of work suffer.
  • An object of the present invention can be seen in providing a technology that is an alternative to expensive GPS systems, in particular, which relieves the driver and offers him more comfort in various applications (e.g. mowing, swathing, loading or baling).
  • a technology which records lane lines (e.g. swaths or mowed edges) in a field, processes corresponding image content, selects the correct lane and then controls the vehicle's steering accordingly.
  • a core of the invention is a system of simple sensors (e.g. cameras) and state-of-the-art image processing (expanded in particular by methods of artificial intelligence, see below), which makes it possible to flexibly design and use the system.
  • the system can be implemented in different ways and thus adapted to customer requirements.
  • the present invention makes it possible to relieve the driver and give him more comfort in various applications (e.g. mowing, swapping den, loading or pressing).
  • the agricultural utility vehicle can thus be used more efficiently. The efficiency increases, work performance and quality of work improve significantly and fuel consumption decreases.
  • a processor unit for the autonomous operation of a vehicle can form an element of an electronic control unit of the vehicle.
  • the processor unit can also form an element of a driver assistance system of the vehicle.
  • the processor unit includes an interface.
  • the interface is in particular a communication interface which enables the exchange of data, specifically an exchange of data between the processor unit on the one hand and a sensor unit on the other.
  • the processor unit can transmit commands to actuators of the vehicle via the interface, for example to a steering actuator of the vehicle.
  • the processor unit is set up in particular to use the interface to access a surrounding area detection of the vehicle generated by the sensor unit, to recognize several lane lines in the surrounding area detection, to select a lane line from the several lane lines and to control the vehicle based on the selected lane line in such a way that the vehicle autonomously follows the selected lane line, e.g. that the vehicle drives on or to the side of the selected track line. If the sensor unit detects only a single lane line in the vicinity of the vehicle, then the processing unit is also set up to recognize this single lane line and to control the vehicle in such a way that it follows it.
  • a driver assistance system for the autonomous operation of a vehicle.
  • the driver assistance system comprises a sensor unit for detecting the surroundings of the vehicle and a processor unit according to the first aspect of the invention.
  • the sensor unit is set up to generate a detection of the surroundings of the vehicle.
  • a corresponding method for the autonomous operation of a vehicle is proposed.
  • the method can generate a detection of the surroundings of the vehicle by a sensor unit and a know several lane lines in the environment detection by means of a processor unit.
  • a lane line can be selected from the multiple lane lines using the processor unit, and the vehicle can be controlled based on the selected lane line using the processor unit so that the vehicle autonomously follows the selected lane line, e.g. the vehicle drives on or to the side of the selected lane line .
  • a computer program product for the autonomous operation of a vehicle is also provided.
  • the computer program product instructs the processor unit to use an interface to access a vehicle environment detection generated by a sensor unit, to recognize several lane lines in the area detection, to select a lane line from the several lane lines, and to control the vehicle based on the selected lane line in such a way that the vehicle autonomously follows the selected lane line, e.g. that the vehicle is driving on or to the side of the selected track line.
  • a vehicle in particular an off-road vehicle.
  • the vehicle can comprise a processor unit according to the first aspect of the invention.
  • the vehicle can include a driver assistance system according to the second aspect of the invention.
  • the vehicle can be controlled by means of the processor unit.
  • “Controlled” or “steer” can in particular be understood to mean that the vehicle can be operated autonomously, ie it can in particular be automatically steered, accelerated and braked, including all of the necessary controls Actuators in particular of the drive train, the steering and signaling of the vehicle.
  • the control of the vehicle by means of the processor unit can be activated automatically or by a user of the vehicle by means of an operating element when the vehicle is traveling along a lane line. If the vehicle turns at a headland of the track line (s), the control of the vehicle can be deactivated automatically or by the user of the vehicle using an operating element.
  • a track line can, for example, be a swath or a mowing edge or a cutting edge on an agricultural area, e.g. a field.
  • the lane line can comprise a lane, a row of plants, a row of trees, a curb or a work path.
  • the processor unit can be set up not only to recognize swaths and mowed edges as lane lines and instruct the vehicle to follow them, but also other applications in agriculture, in the municipal sector or in the construction machinery business can be successfully automated with simple application effort.
  • rows of maize, apple trees in plantations, vines in vineyards, cut edges of a combine harvester in grain, hoed crops (soybeans, sugar beets, etc.), potato ridges or curbs for road sweepers when mowing roadsides can be recognized.
  • the track line can, in particular, be located in a field and, for example, have been caused by previous sowing.
  • Plant rows are rows of the cultivated product in question, e.g. Corn or grapevines. Rows of trees are for example on orchards, e.g. Apple plantations to be found. You can find commutes to work, for example, in mining (especially in open-cast mining).
  • the vehicle can in particular be an off-road vehicle.
  • An off-road vehicle can be understood to mean a vehicle whose primary area of application is not a road (as is the case, for example, with passenger cars, buses or trucks), but for example an agricultural area, e.g. a field to be worked / tilled or a Forest area, or a mining area (especially open-cast mining), or an industrial area, e.g. within a production facility or warehouse.
  • the off-road vehicle can be an agricultural utility vehicle or an agricultural tractor such as a combine harvester or tractor.
  • the vehicle can be an industrial truck, for example a forklift or a tractor.
  • the vehicle can also be a municipal vehicle, for example a street sweeper, a garbage truck or a snow clearing vehicle.
  • the processor unit is set up to select that lane line which has the smallest distance from the vehicle.
  • the detection of the surroundings can comprise several camera images recorded one after the other (image sequence), which depict the surroundings of the vehicle.
  • image sequence For example, a first track line and a second track line can be recognized in one of these camera images.
  • the first track line and the second track line can run parallel to one another and to the side of the vehicle. This can be recognized by methods of image processing. Image processing methods can also be used to identify which of the two lane lines is closer to the side of the vehicle. If this is the first lane line, the processor unit can select the first lane line and control the vehicle in such a way that it drives on or parallel to the first lane line.
  • the processor unit determines that the second lane line has a smaller distance from the vehicle than the first lane line, then it can select the second lane line and control the vehicle in such a way that it drives, for example, on or at a lateral distance parallel to the second lane line .
  • the present invention supports several different applications (e.g., mowing, swathing, loading or baling). Furthermore, the present invention enables track lines to be followed at a certain distance, which is particularly important with different working widths of attachments that can be attached to the vehicle. Furthermore, the present invention enables track lines to be reliably recognized. Also can
  • Swaths and cut edges can be recognized, for example, with a lateral distance of up to 8 m, the agricultural utility vehicle being able to be controlled in such a way that the named recognized objects are followed.
  • the sensor unit is set up to detect a local area surrounding the vehicle.
  • the local surroundings of the vehicle can be referred to as the region of interest (ROI).
  • ROI region of interest
  • the region of interest can be a field or part of a field.
  • the field or the part of the field can be intended to be worked in particular for agriculture by the vehicle.
  • the vehicle or an attachment of the vehicle can have means that enable it to carry out a specific operation within the work area, for example mowing, swathing, loading or baling. These means can be integrated into the vehicle or attached to the vehicle as attachments.
  • the recordings or frames produced when the surroundings are recorded result in the recording of the surroundings.
  • the detection of the surroundings can be images if a camera or a camera system is used as the sensor.
  • the detection of the surroundings can be frames if, for example, a radar or a lidar is used.
  • the recordings, in particular the images or frames each cover a limited area around the vehicle. This is what the “local” feature means. The local area around the vehicle is thus a limited area that extends around the outside of the vehicle.
  • the range or extent of the local environment can vary depending on the type of sensor used and can be adjusted if necessary.
  • the sensor system disclosed in the context of the present application is designed to be arranged on the vehicle, in particular to be attached, in such a way that it can detect the local surroundings of the vehicle.
  • the area of the environment which the relevant sensor detects can also be referred to as the so-called “field of view”. This area can be one-dimensional, two-dimensional or three-dimensional, depending on the sensor used. It is possible that the area that can be detected by the sensor can detect part of the surroundings of the vehicle, for example a sector in the front area and / or in the side area and / or in the rear area of the vehicle.
  • the relevant sensor can also be set up to record the entire surroundings of the vehicle, for example when using so-called surround view systems.
  • the processor unit is set up to evaluate the detection of the surroundings.
  • the detection of the surroundings can, for example, be a recording of the local surroundings of the vehicle.
  • the processor unit is set up to extract objects from recordings or frames generated by the sensor unit.
  • the processor unit can evaluate an image from a camera of the sensor unit and recognize all traces within the image.
  • the multiple lane lines can be recognized by means of image processing methods, for example within an image that has been recorded by a camera of the sensor unit and that depicts the surroundings of the vehicle.
  • image processing methods for example within an image that has been recorded by a camera of the sensor unit and that depicts the surroundings of the vehicle.
  • suitable methods of image processing are, for example, semantic segmentation or edge detection.
  • the result of using such image processing methods can in particular be a polynomial that describes the extent and course of the track lines.
  • the processor unit is set up to control the vehicle based on at least one of the following parameters, namely based on a curvature of the selected lane line, based on a course error of the vehicle or based on a difference from one determined from the surroundings lateral distance of the selected lane line to the vehicle including its attachments and a desired distance from the selected lane line to the vehicle including its attachments.
  • the processor unit is set up to recognize the multiple track lines in the detection of the surroundings by combining methods of image processing with methods of artificial intelligence.
  • An artificial neural network can be trained, for example by making representative sample images with track lines available to the artificial neural network for training it.
  • the extension of the “classic image processing” with elements of artificial intelligence makes it possible in particular to recognize several lane lines and to continuously improve the degree of recognition and thus the reliability by training a neural network, for example.
  • An output through the combination of classic image processing and artificial intelligence Polynomial can correctly describe the curvature of the track line, the course of the track line and the lateral distance between the track line and the off-road vehicle including its attachments with a particularly high probability. Together with the desired distance between the track line and the off-road vehicle, including its attachments (“desired lateral offset”), the vehicle's steering can be controlled.
  • images or frames generated by the sensor unit can be processed by means of so-called semantic segmentation (in English: “Semantic segmentation”), which uses an artificial neural network.
  • semantic segmentation uses an artificial neural network.
  • Adjacent pixels can be combined to form image regions with coherent content if they meet a defined homogeneity criterion.
  • a pixel-precise classification of the images can take place, it being possible for each pixel to belong to a known object class.
  • the processor unit can receive an image sequence from a camera which records the surroundings of the vehicle in successive images of the image sequence.
  • the processor unit can extract features using semantic segmentation, for example, and assign each pixel to an object class in a classification, for example an object class “track line”, an object class “field without track line”, “sky” etc.
  • semantic segmentation method is suitable especially for complex applications and provides a particularly high level of accuracy.
  • images or frames generated by the sensor unit can be processed using the edge detection method that uses an artificial neural network.
  • a filter can be adapted using the artificial neural network.
  • edge detection method flat areas in a digital image can be separated from one another, especially if they differ sufficiently in terms of color or gray value, brightness or texture along straight or curved lines.
  • Special edge operators also known as filters, can recognize transitions between flat areas and mark them as edges.
  • edges are detected in a two-dimensional image, for example, which has been recorded by a camera, for example.
  • Information relating to recognized edges can be used to recognize certain objects in the image, with reference to the present invention in particular for recognizing track lines in a field. For this purpose, for example, the information relating to recognized edges can be compared with stored patterns.
  • the system also enables control or automation of an attachment in the vehicle.
  • a loader wagon pickup (as an example of an attachment) can e.g. be lifted via a CAN signal when the end of a swath is detected by the sensor unit.
  • the loading wagon pick-up can e.g. can be lowered via a CAN signal when the beginning of a swath is detected by the sensor unit.
  • the processor unit is set up to control an attachment mounted on the vehicle based on the selected track line and its course in the environment detection.
  • the desired distance to the track lines (“Desired Lateral Offset") can be defined by the width of the attachment. This can either be entered manually by the driver via a terminal or through automatic attachment recognition, e.g. by means of a tag on the attachment, the tag being recognized and processed directly by the processor unit or the driver assistance system.
  • the processor unit can be integrated into a driver assistance system of the vehicle or be communicatively connected to the driver assistance system.
  • the processor unit can be set up to initiate that the vehicle is transferred to a safe state when the processor unit detects at least one potential collision object in the surroundings detection.
  • a potential collision object can be an object (stationary or movable, for example a tree, a power pole, a pole or another vehicle), a person or an animal that is located within the detected local area surrounding the vehicle.
  • “initiate” can in particular be understood to mean that the processor unit a command is transmitted to the driver assistance system of the vehicle so that the driver assistance system transfers the vehicle to the safe state.
  • the driver assistance system can bring the vehicle to a standstill when it is brought into the safe state.
  • the speed can be reduced (without having to stop the vehicle) so that the risk of collision can be significantly reduced.
  • an optical or acoustic warning can be output, in particular to the user of the vehicle (for example via a display inside the vehicle) and / or to the local area around the vehicle, for example by a horn, a horn, a loudspeaker or by a Vehicle lighting system.
  • the visual or acoustic warning can in particular be perceived by people and animals who are in the vicinity of the vehicle. In this way, people in the vicinity can be warned of the autonomously driving vehicle and collisions can be avoided.
  • the sensor unit for detecting the surroundings of the vehicle can comprise at least one of the following sensors, namely an image processing sensor, in particular a camera, a radar sensor or a laser-based sensor.
  • the image processing sensor e.g. a camera
  • the image processing sensor can be set up to record images of the surrounding area.
  • features can be recognized in the images, in particular the track lines.
  • the radar-based sensor can be set up to recognize features in the detected surroundings of the vehicle, in particular the lane lines.
  • the radar-based sensor can, for example, measure distances to objects within the detected surroundings.
  • the radar-based sensor can also measure azimuth values, altitude values (elevation), intensity values and radial velocity values, for example.
  • the radar sensor can be used to determine the incline.
  • the sensor unit can include at least one camera which detects track lines in a field of view of the camera, and in addition the sensor can Sensor unit comprise a radar-based sensor which is set up to determine the slope of the terrain in a field of view of the radar-based sensor.
  • the fields of view of the camera and the radar-based sensor can be congruent, overlap or not overlap (complementary fields of view).
  • a corresponding measurement cycle in which the radar-based sensor recorded the surroundings of the vehicle or measured it in the manner described can be referred to as a “frame”.
  • the radar-based sensor can therefore scan or detect the environment in an N-dimensional manner, as a result of which point clouds can be generated.
  • the radar-based sensor can extract features from recorded point clouds.
  • the point cloud can accordingly comprise several dimensions (N-dimensional point cloud) if, for example, intensities and radial speeds are also taken into account.
  • the laser-based sensor (e.g. a lidar sensor) can be set up to recognize features in the detected surroundings of the vehicle, in particular the lane lines.
  • the laser-based sensor can, for example, measure intensities in an x-direction, in a y-direction and in a z-direction of a Cartesian coordinate system of the laser-based sensor within the detected surroundings.
  • a corresponding measuring cycle in which the laser-based sensor has recorded the environment or measured it in the described manner can be referred to as a “frame”.
  • the laser-based sensor can scan or detect the environment N-dimensionally, whereby point clouds can be generated.
  • the laser-based sensor can extract features from captured point clouds.
  • the point cloud can accordingly comprise several dimensions (N-dimensional point cloud).
  • a selection of a track line to be followed can be controlled by a driver or user of the agricultural utility vehicle via an operating aid.
  • the system comprises, in a further embodiment, an operating unit which is set up to enable a user to select a track line from the multiple track lines.
  • the operating unit can comprise a display, for example a touch screen.
  • the display can detect the plural ones by the sensor unit Show track lines. The user can select the lane line that the vehicle should follow autonomously from the multiple lane lines shown.
  • FIG. 1 shows a plan view of a field on which a tractor with an attachment drives autonomously
  • FIG. 2 elements of a driver assistance system for the tractor according to FIG. 1,
  • FIG. 3 shows steps of a method for detecting track lines and for controlling the tractor according to FIG. 1 based on a selected track line by means of the driver assistance system according to FIG. 2, and
  • FIG. 4 scenarios A to F, in which the driver assistance system according to FIG. 2 is active or passive in a T raktor.
  • FIG. 1 shows an agricultural machine in the form of a tractor 1.
  • an attachment 2 is attached to the tractor 1.
  • the tractor 1 pulls the attachment over a field 3 so that the attachment 2 can work on the field 3 and so that several track lines are created, in the exemplary embodiment shown, a first track line 4.1 and a second track line 4.2.
  • the first track line 4.1 runs parallel to the second track line 4.2.
  • the tractor 1 is located with the attachment 2 laterally spaced apart from the first track line 4.1.
  • the second track line 4.2 is further away from the vehicle 1 and the attachment 2 than the first track line 4.1.
  • the vehicle 1 comprises a system 5, shown in more detail in FIG. 2, for the autonomous operation of the vehicle 1 (in certain situations / scenarios, see FIG. 4).
  • the system 5 comprises a processor unit 6, a memory unit 7, a communication interface 8 and a sensor unit 9.
  • the sensor unit 9 comprises a digital camera system which is referred to below as “camera” and is provided with the reference number 11 in FIG. 2.
  • the camera 11 is arranged in particular on a roof of the tractor 1 so that it covers an environment 12 of the tractor 1 that is located in a forward area and on both sides of the tractor 1.
  • the digital camera system 1 1 can also, for example, be a
  • Surround view system that can capture an environment 12 that extends 360 ° around the tractor 1.
  • the camera 11 successively and continuously records images of the surroundings 12 of the tractor 1, as a result of which a video or an image sequence of the surroundings 12 of the tractor 1 is generated.
  • This video or this sequence of images is the detection of the surroundings of tractor 1.
  • image sections can be defined (regions of interest, or ROI for short), within which the track lines 4.1 and 4.2 can be recognized using image recognition methods.
  • Methods of artificial intelligence or machine learning can optionally be used.
  • semantic segmentation can be selected as the method of image recognition, whereby an artificial neural network is used (e.g. a convolutional neural network, or CNN for short) that is trained with reference images.
  • the sensor unit 10 can have a radar-based sensor 13.
  • the radar-based sensor 13 is arranged on the tractor 1 in such a way that it can determine altitude values which allow a calculation of a slope of the terrain which is present in the detected surroundings 12 of the tractor 1.
  • a computer program product 10 can be stored on the storage unit 7.
  • the computer program product 10 can be executed on the processor unit 6, for which purpose the processor unit 6 and the memory unit 7 are connected to one another in a correspondingly communicative manner.
  • the computer program product 10 is executed on the processor unit 6, it instructs the processor unit 6 to carry out the functions or method steps described in connection with the drawing.
  • the tractor 1 can be assigned a coordinate system with an x-axis (longitudinal direction) and a y-axis (cross-direction or width direction) running transversely thereto. net that can be viewed as a reference coordinate system.
  • a camera coordinate system can be assigned to each image recorded by the camera 11.
  • Fig. 1 shows that this coordinate system can have a longitudinal axis L of the Ge voltage from tractor 1 and attachment 2.
  • the longitudinal axis L of the image coordinate system coincides with the x-axis of the tractor coordinate system.
  • the image coordinate system also has a width axis B of the attachment 1, which extends transversely to the longitudinal axis L and which extends laterally in width direction B than the tractor 1.
  • a width axis B of the attachment which extends transversely to the longitudinal axis L and which extends laterally in width direction B than the tractor 1.
  • the width axis L does not match the y- Axis of the tractor coordinate system, but there is a longitudinal offset xO in the longitudinal direction x.
  • the processor unit 6 is set up to take this offset into account by converting determined image coordinates into tractor coordinates.
  • the processor unit 6 is also set up to take into account an offset in the width direction y by converting determined image coordinates into tractor coordinates (even if there is no offset in the width direction y in the present example according to FIG. 1).
  • the width axis B can for example run through the widest portion of the attachment 2, as shown by FIG.
  • the wider attachment 2 should have a desired lateral offset (“desired lateral offset”, “target distance”) from a nearest track line (in the exemplary embodiment according to FIG. 1 this is the first track line 4.1), while the combination of tractor 1 and attachment 2 moves on field 3 and works on field 3 by means of attachment 2.
  • This laterally desired distance can be a few meters, for example 8 m.
  • a distance BO can be defined which the closest track line 4.1 should have at least from the longitudinal axis L of the combination of tractor 1 and attachment 2.
  • the desired lateral distance BO can be present as a distance data record to which the processor unit 6 has access.
  • the attachment 2 can comprise a tag, for example an RFID tag, which can be read out by the driver assistance system 5.
  • the distance data record can be stored on the day.
  • the distance data record can also be replaced by a User or a driver of the tractor 1 can be generated in the driver assistance system 5, for example by input via a corresponding terminal of the tractor 1.
  • a second method step 200 the camera 11 records an image of the surroundings of the tractor 1.
  • the second step 200 can be carried out after the first step 100. Alternatively, however, steps 100 and 200 can also be carried out at the same time or with a time overlap.
  • the processor unit 6 can evaluate the image recorded by the camera 11 and recognize the track lines 4.1 and 4.2, whereby methods of artificial intelligence can be used as mentioned above.
  • a polynomial fitting can be carried out.
  • the image content is further processed and polynomials are calculated from individual information (points, broken lines, etc.) that define a track line and its direction.
  • the result of this fourth method step 400 are thus polynomials in the image coordinate system (L, B).
  • the polynomials that have previously been calculated in the image coordinate system can be transformed into coordinates of the tractor coordinate system.
  • the result of this fifth method step 500 are thus polynomials in the tractor coordinate system (x, y).
  • the track lines 4.1 and 4.2 can be swaths or cut edges, for example.
  • the first track line 4.1 has a first lateral distance (first “lateral offset” / first “actual distance”) B1 in the width direction B from the longitudinal axis L of the combination of tractor 1 and attachment 2.
  • the second track line 4.2 has a second lateral distance (second “lateral offset”; second “actual distance”) B2 to the longitudinal axis L of the combination of tractor 1 and attachment 2 in the width direction B.
  • the first lateral distance B1 is smaller than the second lateral distance B2, ie the first track line 4.1 is closer to the longitudinal axis L of the combination of tractor 1 and attachment 2 in the width direction B than the second track line 4.2.
  • step 600 FIG.
  • the processor unit 6 can evaluate the image to determine which of the two track lines 4.1 and 4.2 has a smaller lateral distance (in the width direction B) to the combination of tractor 1 and attachment 2. In the exemplary embodiment according to FIG. 1, the processor unit 6 will determine that the first track line 4.1 has the smaller lateral distance to the combination of tractor 1 and attachment 2 than the second track line 4.2 (B1 ⁇ B2). The processor unit 6 will select the first track line 4.1 accordingly.
  • the processor unit 6 can control the tractor 1 in a seventh method step 700 such that the combination of tractor 1 and attachment 2 moves autonomously along the selected first track line 4.1.
  • the attachment 2 can pursue its main function, namely to work the field 3 and for example generate a new track line next to the first track line 4.1 or collect the first track line 4.1.
  • the first lateral actual distance B1 is smaller than the lateral setpoint distance BO. However, it is required or desired that the actual distance B1 is at least as great as the setpoint distance BO.
  • the processor unit 6 can determine this and, for example, have a corrective effect on the steering of the tractor 1 in such a way that the combination of tractor 1 and attachment 2 moves away from the first track line 4.1 (to the left according to the view in FIG. 1) that the distance B1 of the longitudinal axis L rises from the first track line 4.1 in subsequent images recorded by the camera 11 and that the target criterion B1> BO is met.
  • the steering accuracy provided by the driver assistance system 5 is high and the tractor 1 follows the track line 4.1 perfectly at the distance BO maintained.
  • the track lines 4.1 and 4.2 each have a curvature.
  • the processor unit 6 is set up to determine the curvature of the track lines 4.1 and 4.2. For example, as shown by FIG. 1, the processor unit 6 can apply a tangent T to the first track line 4.1 where the latitude axis B intersects the first track line 4.1. This tangent T can be so be moved so that it lies at the intersection between the longitudinal axis L and the latitude axis B. The processor unit 6 can calculate an angle D between this shifted tangent T ′ and the longitudinal axis L. The angle D can be viewed as a heading error of the tractor 1.
  • the angle D can be caused, for example, by drift, wind, traction problems or centrifugal forces. If the tractor 1 were to continue to move straight ahead in the direction of the longitudinal axis L, then the tractor 1 and the attachment 2 would approach the first track line 4.1 and possibly drive over it. However, in order to follow the curved first track line 4.1 with the lateral target distance BO, the processor unit 6 can, for example, have a corrective effect on the steering of the tractor 1 in such a way that the course of the shifted tangent T is subsequently recorded by the camera 11 Bil countries approximates the course of the longitudinal axis L, and that the course error in the form of the angle D is minimized.
  • the processor unit 6 can furthermore be set up to initiate that the tractor is brought into a safe state (e.g. stopped) when the processor unit 6 detects at least one potential collision object 15 in the detection of the surroundings, e.g. a person, stake, or other vehicle. Alternatively, a warning can also be given to the driver of the actuator 1 or its surroundings 12.
  • a safe state e.g. stopped
  • a warning can also be given to the driver of the actuator 1 or its surroundings 12.
  • the driver assistance system 5 of a tractor 1 can then be active and control the tractor 1 as described above when the tractor 1 is driving on the field 3 along the lane line 4 (scenarios A, B and C). If problems arise with the detection of the lane lines 4, the driver assistance system 5 transfers control of the control of the tractor 1 back to the driver of the tractor 1. Even while the driver is turning the tractor 1 at a headland 14, the driver assistance system 5 transfers control of the control of the tractor 1 back to the driver of the tractor 1.
  • the driver assistance system 5 passes control of the control of the tractor 1 back to the driver of the tractor 1 if the driver does not turn at the headland 14 or takes a completely wrong course (scenarios D, E and F), because in these cases For example, no track line 4 in the vicinity of the tractor 1 can be detected by the sensor unit of the tractor 1 and the conditions for controlling the tractor 1 can therefore not be met.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Mechanical Engineering (AREA)
  • Artificial Intelligence (AREA)
  • Automation & Control Theory (AREA)
  • Software Systems (AREA)
  • Transportation (AREA)
  • Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Soil Sciences (AREA)
  • Environmental Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Traffic Control Systems (AREA)

Abstract

L'invention concerne le fonctionnement autonome d'un véhicule (1). Ainsi, l'invention propose par exemple une unité de processeur servant à faire fonctionner de manière autonome un véhicule (1). L'unité de processeur comprend une interface. L'unité de processeur est mise au point pour recourir au moyen de l'interface à une détection du champ environnant du véhicule (1) générée par une unité de capteur, pour identifier plusieurs lignes (4.1, 4.2) dans la détection de champ environnant, pour sélectionner une ligne (4.1) parmi les plusieurs lignes (4.1, 4.2) et pour commander le véhicule (1) sur la base de la ligne (4.1) sélectionnée de telle manière que le véhicule (1) suit de manière autonome la ligne (4.1) sélectionnée. L'invention concerne par ailleurs un système d'aide au conducteur (5) correspondant, un procédé ainsi qu'un produit-programme informatique et un véhicule (1).
PCT/EP2020/055651 2019-03-11 2020-03-04 Système d'aide au pilotage basé sur la vision pour des véhicules terrestres Ceased WO2020182564A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102019203247.8A DE102019203247A1 (de) 2019-03-11 2019-03-11 Vision-basiertes Lenkungsassistenzsystem für Landfahrzeuge
DE102019203247.8 2019-03-11

Publications (1)

Publication Number Publication Date
WO2020182564A1 true WO2020182564A1 (fr) 2020-09-17

Family

ID=69845334

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2020/055651 Ceased WO2020182564A1 (fr) 2019-03-11 2020-03-04 Système d'aide au pilotage basé sur la vision pour des véhicules terrestres

Country Status (2)

Country Link
DE (1) DE102019203247A1 (fr)
WO (1) WO2020182564A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4085740A1 (fr) * 2021-04-30 2022-11-09 Deere & Company Système de guidage visuel utilisant la détection dynamique des bords
WO2023024516A1 (fr) * 2021-08-23 2023-03-02 上海商汤智能科技有限公司 Procédé et appareil d'avertissement précoce de collision, dispositif électronique et support de stockage
CN116888630A (zh) * 2021-02-24 2023-10-13 大陆智行德国有限公司 检测对象和确定对象高度的方法和装置

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112319468B (zh) * 2020-11-12 2021-07-20 上海伯镭智能科技有限公司 维持路肩间距的无人驾驶车道保持方法
DE102022117447A1 (de) 2022-07-13 2024-01-18 Claas Selbstfahrende Erntemaschinen Gmbh Verfahren zur Bearbeitung eines Feldes mit einer auf dem Feld stehenden Photovoltaikanlage
DE102022121482A1 (de) * 2022-08-25 2024-03-07 Claas Selbstfahrende Erntemaschinen Gmbh System zur Bestimmung einer Bestandskante sowie selbstfahrende Erntemaschine
CN118819138A (zh) * 2024-05-31 2024-10-22 中国矿业大学(北京) 基于视觉的井下辅运机器人iCAR导航方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6385515B1 (en) * 2000-06-15 2002-05-07 Case Corporation Trajectory path planner for a vision guidance system
EP2798928A1 (fr) * 2013-04-29 2014-11-05 CLAAS Agrosystems KGaA mbH & Co KG. Système d'operation et procédé permettant de faire fonctionner un système de guidage automatique d'un véhicule agricole
US20180084708A1 (en) * 2016-09-27 2018-03-29 Claas Selbstfahrende Erntemaschinen Gmbh Agricultural work machine for avoiding anomalies
US20190059199A1 (en) * 2017-08-31 2019-02-28 Cnh Industrial America Llc System and method for strip till implement guidance monitoring and adjustment

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE19743884C2 (de) * 1997-10-04 2003-10-09 Claas Selbstfahr Erntemasch Vorrichtung und Verfahren zur berührungslosen Erkennung von Bearbeitungsgrenzen oder entsprechenden Leitgrößen
DE102006055858A1 (de) * 2006-11-27 2008-05-29 Carl Zeiss Ag Verfahren und Anordnung zur Steuerung eines Fahrzeuges
DE102017117148A1 (de) * 2017-07-28 2019-01-31 RobArt GmbH Magnetometer für die roboternavigation

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6385515B1 (en) * 2000-06-15 2002-05-07 Case Corporation Trajectory path planner for a vision guidance system
EP2798928A1 (fr) * 2013-04-29 2014-11-05 CLAAS Agrosystems KGaA mbH & Co KG. Système d'operation et procédé permettant de faire fonctionner un système de guidage automatique d'un véhicule agricole
US20180084708A1 (en) * 2016-09-27 2018-03-29 Claas Selbstfahrende Erntemaschinen Gmbh Agricultural work machine for avoiding anomalies
US20190059199A1 (en) * 2017-08-31 2019-02-28 Cnh Industrial America Llc System and method for strip till implement guidance monitoring and adjustment

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116888630A (zh) * 2021-02-24 2023-10-13 大陆智行德国有限公司 检测对象和确定对象高度的方法和装置
EP4085740A1 (fr) * 2021-04-30 2022-11-09 Deere & Company Système de guidage visuel utilisant la détection dynamique des bords
US12112546B2 (en) 2021-04-30 2024-10-08 Deere & Company Vision guidance system using dynamic edge detection
WO2023024516A1 (fr) * 2021-08-23 2023-03-02 上海商汤智能科技有限公司 Procédé et appareil d'avertissement précoce de collision, dispositif électronique et support de stockage

Also Published As

Publication number Publication date
DE102019203247A1 (de) 2020-09-17

Similar Documents

Publication Publication Date Title
WO2020182564A1 (fr) Système d'aide au pilotage basé sur la vision pour des véhicules terrestres
DE69814847T2 (de) Agrar-erntemaschine mit roboter-kontrolle
EP3666050B1 (fr) Système de travail agricole
EP3299996B1 (fr) Machines agricoles dotées d'un système de traitement d'images
DE102016209437B4 (de) Selbsttätiges Lenksystem zur Führung eines landwirtschaftlichen Fahrzeugs über ein Feld und entsprechendes Verfahren
DE102008015277A1 (de) Verfahren und Vorrichtung zur Lenkung einer zweiten landwirtschaftlichen Maschine, die relativ zu einer ersten landwirtschaftlichen Maschine über ein Feld lenkbar ist
DE10351861A1 (de) Verfahren und Lenksystem zum selbsttätigen Lenken einer landwirtschaftlichen Maschine
EP4064818B1 (fr) Procédé de traitement de plantes dans un champ
EP4064819B1 (fr) Procédé de traitement de plantes dans un champ, de génération de données d'apprentissage et d'entraînement d'un réseau de neurones
EP3818799B1 (fr) Machine de travail agricole
EP3783529A1 (fr) Système de sécurité et procédé de fonctionnement d'une machine de travail agricole mobile
EP4085741A1 (fr) Procédé de direction pour une machine agricole
EP4220530A1 (fr) Procédé d'aide à la classification de données d'environnement d'une machine de travail agricole
EP1769662A1 (fr) Unité de travail agricole avec un dispositif de travail pour la création d' une colonne d'objets
DE102019111317A1 (de) Autonome landwirtschaftliche Arbeitsmaschine und Verfahren zu deren Betrieb
DE102021125360A1 (de) Vorrichtung und Verfahren zur Bestimmung einer Lenkvorgabe für eine landwirtschaftliche Maschine
DE102024118312A1 (de) Führung und/oder automatisierung von arbeitsfahrzeugen in bezug auf erkannte regionen von interesse in einem arbeitsbereich
DE102019006720A1 (de) Autonomer Betrieb eines Fahrzeugs
EP4449849B1 (fr) Système pour éviter une collision entre une moissonneuse agricole et un véhicule d'accompagnement
WO2020160863A1 (fr) Dispositif de planification d'itinéraire pour une machine agricole en fonction de données de détection et par segmentation d'image
WO2020165055A1 (fr) Dispositif et procédé pour le fonctionnement sûr d'un outil monté de manière réglable sur une machine agricole à fonctionnement autonome
DE102019201977A1 (de) Verfahren zum Ermitteln eines Zustandes einer Feldfrucht-Bepflanzung und landwirtschaftliches Fahrzeug
DE102019202069B4 (de) Verfahren zur Kollisionsvermeidung
EP3732950B1 (fr) Procédé de fonctionnement d'un engin agricole automatique
EP4331332A1 (fr) Système de détermination d'un bord de récolte et recolteuse automotrice

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20711806

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 20711806

Country of ref document: EP

Kind code of ref document: A1