DE102023103473A1 - Method and device for determining an upcoming state of a vehicle - Google Patents

Abstract

A device for determining a state distribution of a temporally preceding state, in particular a temporally preceding position, of a vehicle is described. The device is set up to determine environmental data in relation to the environment of the vehicle. The device is also set up to determine a plurality of state distributions for a corresponding plurality of possible maneuvers of the vehicle on the basis of the environmental data and using a determination unit, wherein the determination unit comprises a plurality of different, machine-learned, output networks for the corresponding plurality of possible maneuvers. Furthermore, the device is set up to operate the vehicle on the basis of at least one of the determined state distributions.

Description

The invention relates to a method and a corresponding device for determining, e.g. for predicting and/or planning, an upcoming state, in particular an upcoming movement path, of a (motor) vehicle.

A vehicle can be designed for automated longitudinal and/or transverse guidance of the vehicle. In this context, environmental data relating to the vehicle's surroundings can be recorded using one or more of the vehicle's environmental sensors. Furthermore, an upcoming movement path of the vehicle can be planned on the basis of the environmental data, e.g. a movement path that enables the vehicle to travel collision-free on a roadway that is currently being used. The vehicle can then be guided automatically along the planned movement path in a reliable and safe manner.

This document deals with the technical task of achieving particularly precise planning of an upcoming state, in particular a movement path, of a (motor) vehicle, in particular in order to increase the reliability, robustness and/or safety of the automated longitudinal and/or lateral guidance of the vehicle.

The problem is solved by each of the independent claims. Advantageous embodiments are described, among other things, in the dependent claims. It is pointed out that additional features of a patent claim dependent on an independent patent claim can form a separate invention without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim, which invention is independent of the combination of all features of the independent patent claim and can be made the subject of an independent claim, a divisional application or a subsequent application. This applies equally to technical teachings described in the description, which can form an invention independent of the features of the independent patent claims.

According to one aspect, a device for determining a state distribution of a temporally and/or spatially forward state, in particular a temporally and/or spatially forward position, of a vehicle is described. The state distribution can comprise a sequence of probability distributions of at least one state variable for a corresponding sequence of forward points in time. The at least one state variable can comprise, in particular be, the position of the vehicle. The state distribution can thus in particular indicate a probability distribution of forward movement paths of the vehicle.

The device is designed to determine environmental data relating to the environment of the vehicle (using one or more environmental sensors of the vehicle). The environmental data can in particular include an image from an environmental camera of the vehicle (which, for example, shows the environment ahead of the vehicle).

The device is further configured to determine a plurality of state distributions for a corresponding plurality of possible maneuvers of the vehicle on the basis of the environmental data and by means of a (machine-trained) determination unit.

• • ein Links-Abbiege-Manöver, bei dem das Fahrzeug an einem vorausliegenden Verkehrsknoten-Punkt links abbiegt;
• • ein Geradeaus-Manöver, bei dem das Fahrzeug an einem vorausliegenden Verkehrsknoten-Punkt geradeaus fährt;
• • ein Rechs-Abbiege-Manöver, bei dem das Fahrzeug an einem vorausliegenden Verkehrsknoten-Punkt rechts abbiegt; und/oder
• • ein Spurwechsel-Manöver, bei dem das Fahrzeug auf eine andere Fahrspur einer mehrspurigen Fahrbahn wechselt.

The possible maneuvers of the multitude of possible maneuvers can differ in relation to the direction of travel at a traffic junction point ahead (e.g. at an intersection). Examples of possible maneuvers are,

• • a left-turn manoeuvre in which the vehicle turns left at an upcoming traffic junction point;
• • a straight-ahead maneuver in which the vehicle drives straight ahead at a traffic junction point ahead;
• • a right-turn manoeuvre in which the vehicle turns right at a traffic junction point ahead; and/or
• • a lane-changing maneuver in which the vehicle changes to another lane on a multi-lane roadway.

The determination unit comprises a plurality of different, machine-trained output networks for the corresponding plurality of possible maneuvers. In other words, the determination unit can comprise a plurality of maneuver-specific (neural) output networks. The individual maneuver-specific output networks of the plurality of output networks can each have been trained in advance on the basis of maneuver-specific training data for the possible maneuver with which the respective maneuver-specific output network is associated.

The determination unit can comprise a (common), machine-trained, base network that is designed to determine a (common) set of features based on the environmental data. The individual maneuver-specific output networks The plurality of output networks can each be designed to determine the respective maneuver-specific state distribution on the basis of the (common) set of features. This can result in a particularly efficient and robust determination of the maneuver-specific state distributions.

The device can further be configured to operate the vehicle on the basis of at least one of the determined state distributions (in particular to guide it longitudinally and/or transversely in an automated manner).

The device can be set up, for example, to determine an upcoming driving maneuver of the vehicle, in particular on the basis of a route planned for the vehicle in a navigation system. The state distribution associated with the upcoming driving maneuver can then be selected from the plurality of state distributions. Furthermore, the vehicle can be operated on the basis of the selected state distribution, in particular automatically guided longitudinally and/or transversely.

A device is thus described that uses several maneuver-specific output networks to determine maneuver-specific state distributions for a vehicle. This enables particularly reliable, robust and safe operation of the vehicle.

According to a further aspect, a (road) motor vehicle (in particular a passenger car or a truck or a bus or a motorcycle) is described which comprises the device described in this document.

According to a further aspect, a method for determining a state distribution of a temporally preceding state, in particular a temporally preceding position, of a (motor) vehicle is described. The method comprises determining (using one or more environmental sensors) environmental data in relation to the environment of the vehicle. Furthermore, the method comprises determining, on the basis of the environmental data and using a (machine-trained) determination unit, a plurality of state distributions for a corresponding plurality of possible maneuvers of the vehicle. The determination unit can comprise a plurality of different, machine-trained, output networks for the corresponding plurality of possible maneuvers.

The method may further comprise operating the vehicle based on at least one of the determined state distributions.

According to a further aspect, a software (SW) program is described. The SW program can be configured to be executed on a processor (e.g. on a control unit of a vehicle) and thereby to carry out the method described in this document.

According to a further aspect, a storage medium is described. The storage medium can comprise a software program which is designed to be executed on a processor and thereby to carry out the method described in this document.

It should be noted that the methods, devices and systems described in this document can be used alone or in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, devices and systems described in this document can be combined with one another in a variety of ways. In particular, the features of the claims can be combined with one another in a variety of ways. Furthermore, features listed in brackets are to be understood as optional features.

• 1 beispielhafte Komponenten eines Fahrzeugs;
• 2 eine beispielhafte Fahrsituation des Fahrzeugs mit mehreren möglichen Fahrmanövern;
• 3 eine beispielhafte, auf maschinellem Lernen basierte, Ermittlungseinheit mit einer Vielzahl von Manöver-spezifischen, neuronalen, Ausgangsnetzen; und
• 4 ein Ablaufdiagramm eines beispielhaften Verfahrens zur Prädiktion und/oder zur Planung eines vorausliegenden Zustands eines Fahrzeugs.

The invention is described in more detail below using exemplary embodiments.

• 1 exemplary components of a vehicle;
• 2 an example driving situation of the vehicle with several possible driving maneuvers;
• 3 an exemplary machine learning-based detection unit with a variety of maneuver-specific neural output networks; and
• 4 a flow chart of an exemplary method for predicting and/or planning an upcoming state of a vehicle.

As stated at the beginning, this document deals with the reliable and precise estimation of the forward state of a (motor) vehicle. The forward state can indicate the position of the vehicle as a function of time, i.e. the forward movement path of the vehicle, along a sequence of points in time.

1 an exemplary vehicle 100 having one or more environmental sensors 102, each configured to collect sensor data (also referred to as environmental data) relating to the environment of the vehicle 100. Example environmental sensors 102 are a camera, a radar sensor, a lidar sensor, an ultrasonic sensor, etc.

A (control) device 101 of the vehicle 100 can be set up to evaluate the environmental data, e.g. to detect one or more objects (e.g. other vehicles) in the environment of the vehicle 100. The device 101 can also be set up to effect an automated longitudinal and/or transverse guidance of the vehicle 100 based on the environmental data, in particular based on the one or more detected objects. For this purpose, one or more longitudinal and/or transverse guidance actuators 103 (e.g. a drive motor, a braking device and/or a steering device) of the vehicle can be controlled.

The device 101 can in particular be set up to determine (i.e. to plan and/or estimate) an upcoming state of the vehicle 100 as a function of time based on the environmental data, e.g. based on an image captured by a camera 102. The state can comprise one or more state variables, such as the position, the driving speed, the acceleration and/or the orientation. In the following, the upcoming movement path of the vehicle 100 (i.e. the upcoming position of the vehicle 100 as a function of time) will be discussed in particular. It should be noted, however, that the aspects described in this document are generally applicable to the upcoming vehicle state as a function of time.

The determined movement path (generally the determined temporal state progression) can be used as a basis for controlling one or more longitudinal and/or lateral guidance actuators 103 in order to bring about a particularly reliable and precise automated longitudinal and/or lateral guidance of the vehicle 100.

The determination of a motion path can be achieved on the basis of a neural network that has been trained in advance. An example method for determining a motion path is described in Zeng, Wenyuan, et al. "End-to-end interpretable neural motion planner." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. This document is incorporated in its entirety into this document by reference.

It often happens that a vehicle 100 is in a driving situation in which the vehicle 100 can perform several different driving maneuvers, wherein the movement path actually realized by the vehicle 100 depends on the respective driving maneuver. 2 shows an example driving situation in which the vehicle 100 is driving towards a traffic junction 200, in particular towards an intersection. The vehicle 100 can carry out several different maneuvers 202 at the traffic junction 200, e.g. a left-turn maneuver, a straight-ahead maneuver or a right-turn maneuver.

The quality of the determination (in particular the planning) of a movement path for the vehicle 100 can be increased by incorporating knowledge of one or more possible driving maneuvers 202 into the determination of the movement path.

3 illustrates an exemplary determination unit 300 that can be used to determine a movement path of the vehicle 100. The determination unit 300 is based on machine learning. In particular, the determination unit 300 comprises neural networks 301, 302 that were trained in advance on the basis of training data (eg using a backpropagation algorithm).

The determination unit 300 comprises a base network 301, which is designed to determine a set 313 of features based on current environmental data 311 (e.g. based on a current image of a camera 102 of the vehicle 100). Furthermore, the determination unit 300 comprises a plurality of output networks 302 for a corresponding plurality of possible driving maneuvers 202 of the vehicle 100. The individual output networks 302 can each have been specifically trained for the driving maneuver 202 with which the respective output network 302 is associated. For the 2 In the example shown, for example, an output network 302 can be used for left-turn maneuvers, an output network 302 for straight-ahead maneuvers, and an output network 302 for right-turn maneuvers.

An output network 302 is designed to determine a statistical distribution 312 based on the set 313 of features, which, for example, indicates the position of the vehicle 100 as a function of time or generally the state of the vehicle 100 as a function of time. For example, the distribution 312 can indicate a probability distribution of the position of the vehicle 100 for each point in time in a sequence of preceding points in time.

Thus, using the determination unit 300 on the basis of current environmental data 311, a plurality of state (in particular position) distributions 312 of the vehicle 100 can be determined for a corresponding plurality of different maneuvers. ver hypotheses 202 can be determined. In this way, the future state of the vehicle 100, in particular a future movement path of the vehicle 100, can be determined in a particularly precise manner.

For example, the device 101 can be set up to determine (e.g. based on a route already planned in the navigation system of the vehicle 100) the driving maneuver 202 that the vehicle 100 will perform. The distribution 312 associated with this driving maneuver 202 can then be selected from the plurality of state (in particular position) distributions 312 of the vehicle 100 in order to effect the automated longitudinal and/or transverse guidance of the vehicle 100.

To train a maneuver-specific output network 302 for a specific maneuver 202 or for a specific maneuver hypothesis (i.e. for a specific maneuver type), training data can be used in which vehicles 100 have performed the specific maneuver 202. In this way, a particularly precise and robust (monitored) training of the individual maneuver-specific output networks 302 can be achieved.

The use of a plurality of maneuver-specific output networks 302 makes it possible to provide maneuver-specific state distributions 312.

In particular, a semantic meaning (in the form of the respective maneuver hypothesis) can be assigned to the individual state distributions 312. In this way, the quality of the further use of the state distributions 312 for automated longitudinal and/or lateral guidance can be increased.

4 shows a flow chart of a (possibly computer-implemented) method 400 for determining a state distribution 312 of a temporally advanced state, in particular a temporally advanced position, of a (motor) vehicle 100. The state distribution 312 can comprise a temporal sequence of probability distributions for one or more state variables (eg for the position) for a corresponding sequence of future points in time (eg over a temporal horizon of 1 second or more, or 5 seconds or more).

The method 400 includes determining 401 environmental data relating to the environment of the vehicle 100 (using one or more environmental sensors 102 of the vehicle 100).

Furthermore, the method 400 comprises determining 402, on the basis of the environmental data and using a determination unit 300, a plurality of state distributions 312 for a corresponding plurality of possible maneuvers 202 of the vehicle 100. The different possible maneuvers 202 (i.e. the different maneuver types) can differ with respect to the direction of travel of the vehicle 100 at a junction 200.

The (machine-trained) determination unit 300 can comprise a plurality of different, machine-trained, output networks 302 for the corresponding plurality of possible maneuvers 202 (i.e., maneuver types). Each output network 302 can be specifically trained for the respective possible maneuver 202 (i.e., for the respective maneuver type).

Furthermore, the method 400 comprises operating 403 the vehicle 100 on the basis of at least one of the determined state distributions 312. In particular, the automated longitudinal and/or transverse guidance of the vehicle 100 can be effected on the basis of at least one of the determined state distributions 312.

By using maneuver-specific output networks 302, the state distribution 312 for a vehicle 100 can be determined, in particular predicted and/or planned, in a particularly precise and robust manner, thereby increasing the quality of the automated longitudinal and/or lateral guidance of the vehicle 100.

The present invention is not limited to the embodiments shown. In particular, it should be noted that the description and the figures are intended only to illustrate the principle of the proposed methods, devices and systems by way of example.

Claims (10)

Device (101) for determining a state distribution (312) of a temporally preceding state, in particular a temporally preceding position, of a vehicle (100); wherein the device (101) is set up to - determine environmental data in relation to an environment of the vehicle (100); - on the basis of the environmental data and using a determination unit (300), determine a plurality of state distributions (312) for a corresponding plurality of possible maneuvers (202) of the vehicle (100); wherein the determination unit (300) has a plurality of different, machine-learned, output networks (302) for the corresponding plurality of possible maneuvers. vern (202); and - to operate the vehicle (100) on the basis of at least one of the determined state distributions (312). Device (101) according to Claim 1 , wherein - the determination unit (300) comprises a machine-trained base network (301) which is designed to determine a set (313) of features on the basis of the environmental data; and - the individual maneuver-specific output networks (302) of the plurality of output networks (302) are each designed to determine the respective maneuver-specific state distribution (312) on the basis of the set (313) of features. Device (101) according to one of the preceding claims, wherein the individual maneuver-specific output networks (302) of the plurality of output networks (302) were each learned in advance on the basis of maneuver-specific training data for the possible maneuver (202) with which the respective maneuver-specific output network (302) is associated. Device (101) according to one of the preceding claims, wherein a state distribution (312) comprises a sequence of probability distributions of at least one state variable for a corresponding sequence of preceding points in time. Device (101) according to Claim 4 , wherein - the at least one state variable comprises a position of the vehicle (100); and/or - the state distribution (312) indicates a probability distribution of upcoming movement paths of the vehicle (100). Device (101) according to one of the preceding claims, wherein the environment data comprise an image of an environment camera (102) of the vehicle (100). Device (101) according to one of the preceding claims, wherein the possible maneuvers (202) of the plurality of possible maneuvers (202) each differ with respect to a direction of travel at a traffic junction point (200) ahead. Device (101) according to one of the preceding claims, wherein the plurality of possible maneuvers (202) comprises, - a left-turn maneuver in which the vehicle (100) turns left at an upcoming traffic junction point (200); - a straight-ahead maneuver in which the vehicle (100) drives straight ahead at an upcoming traffic junction point (200); - a right-turn maneuver in which the vehicle (100) turns right at an upcoming traffic junction point (200); and/or - a lane-changing maneuver in which the vehicle (100) changes to another lane of a multi-lane roadway. Device (101) according to one of the preceding claims; wherein the device (101) is configured to - determine an upcoming driving maneuver of the vehicle (100), in particular on the basis of a route planned in a navigation system; - select the state distribution (312) associated with the upcoming driving maneuver from the plurality of state distributions (312); and - operate the vehicle (100) on the basis of the selected state distribution (312), in particular to guide it longitudinally and/or transversely in an automated manner. Method (400) for determining a state distribution (312) of a temporally preceding state, in particular a temporally preceding position, of a vehicle (100); wherein the method (400) comprises, - determining (401) environmental data relating to an environment of the vehicle (100); - determining (402), on the basis of the environmental data and using a determination unit (300), a plurality of state distributions (312) for a corresponding plurality of possible maneuvers (202) of the vehicle (100); wherein the determination unit (300) comprises a plurality of different, machine-learned, output networks (302) for the corresponding plurality of possible maneuvers (202); and - operating (403) the vehicle (100) on the basis of at least one of the determined state distributions (312).

Images