EP4384892A1 - Procédé de représentation d'un environnement arrière d'une plateforme mobile accouplée à une remorque - Google Patents

Procédé de représentation d'un environnement arrière d'une plateforme mobile accouplée à une remorque

Info

Publication number
EP4384892A1
EP4384892A1 EP22758439.8A EP22758439A EP4384892A1 EP 4384892 A1 EP4384892 A1 EP 4384892A1 EP 22758439 A EP22758439 A EP 22758439A EP 4384892 A1 EP4384892 A1 EP 4384892A1
Authority
EP
European Patent Office
Prior art keywords
image
mobile platform
trailer
environment
facing camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22758439.8A
Other languages
German (de)
English (en)
Inventor
Jose Domingo Esparza Garcia
Raphael Cano
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Robert Bosch GmbH
Original Assignee
Robert Bosch GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Robert Bosch GmbH filed Critical Robert Bosch GmbH
Publication of EP4384892A1 publication Critical patent/EP4384892A1/fr
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/22—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
    • B60R1/23—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
    • B60R1/26—Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view to the rear of the vehicle

Definitions

  • the automation of driving goes hand in hand with equipping vehicles with increasingly extensive and powerful sensor systems for detecting the surroundings.
  • machine learning methods are used for classification and detection tasks.
  • convolutional neural networks are used.
  • a further camera system in particular a fifth one, which is mechanically coupled to a rear side of the trailer and is oriented opposite to the direction of travel in order to image the area covered by the trailer.
  • this fifth camera the safety of the combination of towing vehicle and trailer can be significantly improved, especially when images from this fifth camera supplement the covered part of the rear camera of the towing vehicle.
  • the fifth camera cannot replace the rear camera of the towing vehicle, especially when maneuvering backwards.
  • 50% of the image of the rear view camera that the trailer takes is obscured.
  • This 50% of the picture can be replaced by pictures of the fifth Camera can be added if the corresponding section of the images from the fifth camera is projected onto the covered area of the rear camera and, to a certain extent, makes the trailer appear transparent.
  • a method for displaying a rear environment of a mobile platform coupled to a trailer a system for displaying a rear environment of a mobile platform coupled to a trailer, a trained neural network, a controller, a Mobile platform and a computer program proposed according to the features of the independent claims.
  • Advantageous configurations are the subject of the dependent claims and the following description.
  • a method for displaying a rear environment of a mobile platform is proposed, the mobile platform being coupled to a trailer and having a first rear-facing camera.
  • the procedure for rendering a rear environment includes the following steps. In one step, a first rear image is provided by the first rear-facing camera.
  • a second rear image is provided, which was generated by a second rear-facing camera, the second rear-facing camera in particular having a different perspective of the rear environment than the first rear-facing camera in order to display the rear environment more completely .
  • a trailer image area is determined in the first rear image, in which part of the environment is covered by the coupled trailer.
  • at least part of the trailer image area in the first image is replaced with a partial image area of the second image in order to represent the rear environment of the mobile platform.
  • the first rearward-facing camera can be a rear camera of the towing vehicle.
  • this method can be used to make the tag “transparent” over a wide range of bending angles, since the tag can be identified in the first rear image, for example using semantic segmentation, and can be replaced with partial areas of the second rear image and thus ensures a complete mapping of the rear environment of the mobile platform.
  • Semantic segmentation can become sufficiently efficient and fast with dedicated hardware to perform real-time semantic segmentation.
  • the neural network must be trained to perform a semantic segmentation of the first back image.
  • the neural network can be trained with reference images of standard trailer types from different perspectives that correspond to the corresponding curve situations.
  • each trailer pixel may be replaced with the video information provided by the second rear-view camera, which may be mechanically coupled to the rear of the trailer.
  • the contour of the trailer can be recognized not only when the trailer is used in a straight line, but also when the towing vehicle turns and there is an articulation angle between the towing vehicle and the trailer.
  • the trailer image area can be replaced with the current video from the trailer's rear camera.
  • a user can “zoom in” into the partial image area, for example using a touchscreen, if this image area is small, in order to have a better overview of the current situation. This achieves an optimal resolution of the camera images.
  • the picture in the picture corresponding to a "transparent” trailer, can be carried out under all articulation angles, so that the "transparent functionality" can be provided in all parking or driving situations with the trailer.
  • the trailer has the second rear-facing camera.
  • the mobile platform has the second rear-facing camera with a different rear-facing angle of view than the first rear-facing camera.
  • the second rear-facing camera is a side camera of the mobile platform.
  • the method can also be used when the first rear image is generated by side cameras.
  • the trailer image area can then also be replaced by partial image areas of the second rear image in order to make the trailer “transparent” in a representation of the rear environment make. For this it may be necessary to train the neural network with reference images that were generated and labeled for this situation.
  • the second rear-facing camera is arranged on an exterior mirror of the mobile platform in order to visualize sides of the trailer, in particular including a rear-facing side view during a turning process.
  • the trailer image area is determined using a trained machine learning system.
  • machine learning systems are a neural convolution network, possibly in combination with fully connected neural networks, possibly using classic regularization and stabilization layers such as batch normalization and training drop-outs, using various activation functions such as Sigmoid and ReLu, etc., classic approaches such as support vector machines, boosting, decision trees and random forests can also be used as machine learning systems for the method described.
  • the signal at a connection of artificial neurons can be a real number, and the output of an artificial neuron is calculated by a non-linear function of the sum of its inputs.
  • the connections of the artificial neurons typically have a weight that adjusts as learning progresses. Weight increases or decreases the strength of the signal on a connection.
  • Artificial neurons can have a threshold such that a signal is only output if the total signal exceeds this threshold.
  • a large number of artificial neurons are combined in layers. Different layers may perform different types of transformations on their inputs. Signals travel from the first layer, the input layer, to the last layer, the output layer; possibly after going through the shifts several times.
  • the architecture of such an artificial neural network can be a neural network which, corresponding to a multi-layer perceptron (MLP), is optionally expanded with further, differently structured layers.
  • MLP multi-layer perceptron
  • MLP Perceptron
  • a deep neural network can have many such intermediate layers.
  • the trained machine learning system is a trained neural network for semantic segmentation of the first back-image.
  • a method for generating a trained neural network for semantically segmenting objects of a digital first rear image of a rear environment of a mobile platform with a plurality of training cycles is proposed, each training cycle having the following steps.
  • a digital first rear image of a rear environment of a mobile platform with at least one fob coupled to the mobile platform is provided.
  • a reference image assigned to the digital first back image is provided, with the at least one tag being labeled in the reference image.
  • the digital first rear image is made available to the neural network as an input signal.
  • the neural network is adapted in order to minimize a deviation in the classification from the respective associated reference image during the semantic segmentation of the at least one tag of the digital first rear image.
  • the neural network is thus trained on standard trailer types and under different rearview angles during the cornering situation.
  • the neural network can be a neural convolution network.
  • Reference images are images that are recorded specifically for teaching a machine learning system and are selected manually, for example and annotated or synthetically generated and where the majority of the domains are labeled in relation to the classification of the domains.
  • a labeling of the areas can be done manually according to the specifications of the classification, such as the at least two blindness attributes.
  • Each neuron of the corresponding architecture of the neural network receives z. B. a random starting weight. Then the input data is fed into the network, and each neuron can weight the input signals with its weight and passes the result on to the neurons of the next layer. The overall result is then provided at the output layer. The size of the error can be calculated, as well as the contribution each neuron made to that error, and then change the weight of each neuron in the direction that minimizes the error. Then recursively runs, re-measures the error and adjusts the weights until an error criterion is met.
  • Such an error criterion can be, for example, the classification error on a test data set, or the current value of a loss function, for example on a training data set.
  • the error criterion can relate to a termination criterion as a step in which overfitting would occur during training or the time available for training has expired.
  • the optical image is provided in digital form as an input signal to the trained neural network.
  • the first back image can be semantically segmented in order to identify a pixel subregion in the first back image in which the tag is mapped. This portion can then be replaced with a corresponding portion of the second rear image to represent the rear surroundings of the mobile platform.
  • the method can be carried out with a plurality of camera systems, in that the images from the camera systems are merged.
  • Control of an at least partially automated vehicle is provided; and/or a warning signal for warning a vehicle occupant is provided based on the first and/or the second error signal.
  • control signal is provided based on a representation of a rear environment of a mobile platform generated according to one of the methods described above. It is to be understood that any determination or calculation of a control signal is used depending on the representation of the rear environment of the mobile platform, although this does not preclude other input variables from being used for this determination of the control signal. This applies accordingly to the provision of a warning signal.
  • a control signal may be provided when the designated trailer image area exits specified boundaries in the first rear image.
  • a system for displaying a rearward environment of a mobile platform coupled to a trailer including a first rearward-facing camera and a second rearward-facing camera. Furthermore, the system has a device for data processing in order to generate a representation of the rear environment of the mobile platform, with a first input for signals from the first rear-facing camera and a second input for signals from the second rear-facing camera and a computing unit and/or a system On-chip, the device for data processing having an output for providing the display of the rear environment, and the processing unit and/or the system-on-chip being set up to carry out one of the methods described above for displaying a rear environment.
  • a neural network is proposed that has been trained according to one of the methods described above.
  • a control device for use in a vehicle which has a device for data processing in order to generate a representation of the rear environment of the mobile platform.
  • the control unit has a first input for signals from the first rear-facing camera and a second input for signals from the second rear-facing camera. Furthermore, the control unit has an arithmetic unit and/or a system-on-chip and an output for WO 2023/016798. g . PCT/EP2022/070991
  • the method for displaying a rear environment can easily be integrated into different systems.
  • a mobile platform in particular an at least partially automated vehicle, is proposed, which has a control unit as described above.
  • a computer program which comprises instructions which, when the computer program is executed by a computer, cause the computer to carry out the method for displaying a rear environment described above.
  • Such a computer program enables the method described to be used in different systems.
  • a mobile platform can be understood to mean an at least partially automated system that is mobile and/or a driver assistance system.
  • An example can be an at least partially automated vehicle or a vehicle with a driver assistance system. That is, in this context, an at least partially automated system includes a mobile platform in terms of at least partially automated functionality, but a mobile platform also includes vehicles and other mobile machines including driver assistance systems.
  • Other examples of mobile platforms can be driver assistance systems with multiple sensors, mobile multi-sensor robots such as e.g.
  • FIG. 1 a shows a first rear image from the first rear-facing camera, the towing vehicle being mechanically coupled to the trailer; b) corresponds to la) wherein a tag image area of the first back image that depicts the tag has been determined;
  • FIG. 2 a shows a first rear image from the first rear-facing camera, the towing vehicle being mechanically coupled to the trailer with a large articulation angle; b) corresponds to 2a) wherein a tag image area of the first back image depicting the tag has been determined;
  • FIG. 3 a shows a first rear image from rear-facing side cameras, the towing vehicle being mechanically coupled to the trailer; b) corresponds to 3a) wherein a tag image area of the first back image depicting the tag has been determined.
  • FIG. 1a schematically sketches a first rear image, which is provided by a first rear-facing camera.
  • the towing vehicle 120 is mechanically coupled to the trailer 100, with the trailer 100 being aligned straight with an articulation angle 110 of 0° in the illustration.
  • Figure 1b schematically sketches how the trailer image area 105 of the first rear image according to Figure la, which depicts the trailer 100, was determined and marked with a pattern in order to be able to replace it with a partial image area of the second rear image .
  • FIG. 2a schematically outlines a first rear image in a second scene, which is provided by a first rear-facing camera.
  • the towing vehicle 120 is mechanically coupled to the trailer 100, with the trailer 100 being pulled by the towing vehicle with a large articulation angle 110 in the illustration.
  • the trailer occupies a larger trailer image area in the first rear image, the area behind the mobile platform can only be seen to a very limited extent.
  • FIG. 2b schematically sketches how the trailer image area 105 of the first rear image corresponding to FIG. 2a, which depicts the trailer 100, was determined and marked with a pattern in order to be able to replace it with a partial image area of the second rear image.
  • FIG. 3a schematically outlines a representation of a rear environment of a mobile platform that was provided and generated with two rear-facing side cameras. Since the towing vehicle is cornering to the left, the image is divided in such a way that a first rear image from the left side camera occupies a larger image portion 320 than a first rear image from the right side camera 310.
  • the towing vehicle has a left outer surface 120a and a right outer surface 120b can be seen.
  • the towing vehicle is mechanically coupled to the trailer 100a with a large articulation angle. Again, the fob 100a blocks a view of a full rear environment of the mobile platform.
  • FIG. 3b schematically sketches how the trailer image area 105 of the first rear image of the left side camera according to FIG. 3a, which images the trailer 100a, was determined and marked with a pattern in order to provide you with a partial image area of the second rear to replace the image.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne un procédé permettant de représenter un environnement arrière d'une plateforme mobile, qui est accouplée à une remorque, la plateforme mobile présentant une première caméra orientée vers l'arrière, ledit procédé comprenant les étapes suivantes : fournir une première image arrière de la première caméra orientée vers l'arrière; fournir une deuxième image arrière, qui a été générée au moyen d'une deuxième caméra orientée vers l'arrière, déterminer une zone d'image de remorque dans la première image arrière, dans laquelle une partie de l'environnement est masquée par la remorque accouplée, et remplacer au moins une partie de la zone d'image de la remorque dans la première image par une zone d'image partielle de la deuxième image arrière, de manière à représenter l'environnement arrière de la plateforme mobile.
EP22758439.8A 2021-08-12 2022-07-26 Procédé de représentation d'un environnement arrière d'une plateforme mobile accouplée à une remorque Pending EP4384892A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102021208825.2A DE102021208825A1 (de) 2021-08-12 2021-08-12 Verfahren zur Darstellung einer rückwärtigen Umgebung einer mobilen Plattform, die mit einem Anhänger gekoppelt ist
PCT/EP2022/070991 WO2023016798A1 (fr) 2021-08-12 2022-07-26 Procédé de représentation d'un environnement arrière d'une plateforme mobile accouplée à une remorque

Publications (1)

Publication Number Publication Date
EP4384892A1 true EP4384892A1 (fr) 2024-06-19

Family

ID=83059275

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22758439.8A Pending EP4384892A1 (fr) 2021-08-12 2022-07-26 Procédé de représentation d'un environnement arrière d'une plateforme mobile accouplée à une remorque

Country Status (5)

Country Link
US (1) US20250085826A1 (fr)
EP (1) EP4384892A1 (fr)
CN (1) CN117836747A (fr)
DE (1) DE102021208825A1 (fr)
WO (1) WO2023016798A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20250278816A1 (en) * 2024-03-04 2025-09-04 Adobe Inc. Custom image and concept combiner using diffusion models

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10701299B2 (en) * 2014-08-18 2020-06-30 Jaguar Land Rover Limited Display system and method
DE102015214611A1 (de) 2015-07-31 2017-02-02 Conti Temic Microelectronic Gmbh Verfahren und Vorrichtung zum Anzeigen einer Umgebungsszene eines Fahrzeuggespanns
DE102017113469A1 (de) 2017-06-20 2018-12-20 Valeo Schalter Und Sensoren Gmbh Verfahren zum Betreiben einer Parkassistenzvorrichtung eines Kraftfahrzeugs mit kombinierter Ansicht aus transparentem Anhänger und Umgebung
DE102019209560A1 (de) * 2019-06-28 2020-12-31 Robert Bosch Gmbh Vorrichtung und Verfahren zum Trainieren eines neuronalen Netzwerks

Also Published As

Publication number Publication date
DE102021208825A1 (de) 2023-02-16
US20250085826A1 (en) 2025-03-13
WO2023016798A1 (fr) 2023-02-16
CN117836747A (zh) 2024-04-05

Similar Documents

Publication Publication Date Title
EP3292510B1 (fr) Procédé et dispositif pour identifier et évaluer des réflexions sur une voie de circulation
WO2022128014A1 (fr) Correction d'images d'un système de caméra panoramique en conditions de pluie, sous une lumière incidente et en cas de salissure
DE102017108254B4 (de) Rundumsichtkamerasystem zur Objekterkennung und -verfolgung und Verfahren zum Ausstatten eines Fahrzeugs mit einem Rundumsichtkamerasystem
EP3532983B1 (fr) Procédé et dispositif pour créer une vue de l'environnement d'un véhicule
DE102020130507A1 (de) Verfahren und vorrichtung zur bestimmung eines gelenkwinkels in einem kraftfahrzeug
DE102018100909A1 (de) Verfahren zum Rekonstruieren von Bildern einer Szene, die durch ein multifokales Kamerasystem aufgenommen werden
EP3293971B1 (fr) Procédé de mise à jour d'une section d'image
DE102019122086A1 (de) Fahrerassistenz für eine Kombination
DE102019218349A1 (de) Verfahren zum Klassifizieren von zumindest einem Ultraschallecho aus Echosignalen
EP3610643A1 (fr) Système de vue panoramique pour un véhicule
EP4384892A1 (fr) Procédé de représentation d'un environnement arrière d'une plateforme mobile accouplée à une remorque
DE102022214210A1 (de) Verfahren zur Darstellung einer erweiterten Ansicht der Umgebung eines Fahrzeugs, Computerprogramm, Rechenvorrichtung und Fahrzeug
DE102022202827A1 (de) Verfahren zum Bestimmen einer Orientierung einer Kamera
WO2022018191A1 (fr) Procédé de détection de dégradation d'imagerie d'un capteur d'imagerie
WO2022042903A1 (fr) Procédé d'identification d'objets tridimensionnels, programme informatique, support de stockage lisible par machine, unité de commande, véhicule et système de surveillance vidéo
DE102018130229B4 (de) Verfahren und Vorrichtung zur Objektextraktion aus eine dreidimensionale Szene darstellenden Szenenbilddaten
DE102020108817A1 (de) Verfahren und System zum Führen eines Fahrzeugs
DE102019102672A1 (de) Intersensorisches lernen
DE102022119751A1 (de) Bestimmen eines Bereichs von Interesse aus Kamerabildern
DE102017221381A1 (de) Verfahren, Vorrichtung und Computerprogramm zum Ermitteln eines Abstandes zu einem Objekt
DE102021208819A1 (de) Verfahren zum Bestimmen eines Knickwinkels
WO2021245156A1 (fr) Classification de données d'image avec ajustement du degré de granulation
WO2021191120A1 (fr) Procédé de détermination d'une valeur d'une variable de contrôleur
DE102020211896A1 (de) Verfahren zur Erzeugung eines Bildes einer Fahrzeugumgebung und Surroundview-System zur Erzeugung eines Bildes einer Fahrzeugumgebung
DE102020130646B4 (de) Verfahren, System und Computerprogrammprodukt zur Erkennung einer Änderung eines Radlenkwinkels bei einem Kraftfahrzeug

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20240312

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)