EP2254104A2 - Procédé de reconnaissance automatique d'une modification de situation - Google Patents

Procédé de reconnaissance automatique d'une modification de situation Download PDF

Info

Publication number
EP2254104A2
EP2254104A2 EP10004126A EP10004126A EP2254104A2 EP 2254104 A2 EP2254104 A2 EP 2254104A2 EP 10004126 A EP10004126 A EP 10004126A EP 10004126 A EP10004126 A EP 10004126A EP 2254104 A2 EP2254104 A2 EP 2254104A2
Authority
EP
European Patent Office
Prior art keywords
behavior
values
field
objects
property
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP10004126A
Other languages
German (de)
English (en)
Other versions
EP2254104B1 (fr
EP2254104A3 (fr
Inventor
Frederik Meysel
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Deutsches Zentrum fuer Luft und Raumfahrt eV
Original Assignee
Deutsches Zentrum fuer Luft und Raumfahrt eV
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Deutsches Zentrum fuer Luft und Raumfahrt eV filed Critical Deutsches Zentrum fuer Luft und Raumfahrt eV
Publication of EP2254104A2 publication Critical patent/EP2254104A2/fr
Publication of EP2254104A3 publication Critical patent/EP2254104A3/fr
Application granted granted Critical
Publication of EP2254104B1 publication Critical patent/EP2254104B1/fr
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/04—Detecting movement of traffic to be counted or controlled using optical or ultrasonic detectors
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions

Definitions

  • the invention relates to a method for automatic detection of a situation change within a spatial area to be monitored by means of sensors.
  • the invention also relates to a device for this purpose.
  • the most commonly used sensors for monitoring are cameras that record the area to be monitored.
  • the captured image of a camera is then forwarded in most cases to a central, where it is then displayed on a screen for an operator.
  • a central where it is then displayed on a screen for an operator.
  • For the operator there is the problem that with an increase in the number of cameras and the number of images that need to be monitored, are increased. Ultimately, this leads to increased complexity, which in the worst case is at the expense of safety.
  • US 2003/0107650 A1 a monitoring and security system which automatically issues a warning when corresponding events that were previously programmed occur in the monitoring of an area.
  • this system can alert you to possible shoplifting if a person opens a plastic bag inside the area to be monitored (here a shop).
  • the system is set up in such a way that it picks up the noise within the store and examines it for the characteristic acoustic pattern of an opening plastic bag.
  • So z. B. in the US 2004/0130620 Al discloses a method for video analysis, in which a plurality of image sensors record an overlapping area, the objects within the surveillance area moving objects can be tracked using a corresponding tracking algorithm, even across the recording limits of an image sensor.
  • a behavior of the objects is then determined for at least one of the fields at least one of its object properties is learned by evaluating the values of the object property stored in the field by means of statistical methods. This results in a corresponding behavior of the objects with respect to the monitored property of the objects for each field of the entire spatial area.
  • a behavior in terms of speed can be learned what z. B. expressed in terms of average speed.
  • the learned behavior of at least one field is compared with currently determined values of the corresponding field, whereby the comparison is based on a situation change is closed.
  • the behavior in step b) is learned by the last ascertained values of the corresponding field, which lie within a certain period of time. So it is z. B. conceivable that only those values are taken into account when learning the behavior that within the last z. For example, five minutes were recorded. This makes it possible for the system to re-learn the changed situation after a certain time and to view it as typical behavior.
  • a behavior is learned for a plurality of discrete periods, which is the basis for the comparison for recognizing a situation change in step c). It is conceivable that a behavior of a period that lies farther back in the comparison is given a lower weighting than those periods that were not so long ago, so that young periods receive a higher weighting than older ones.
  • a position indication is determined which represents the corresponding spatial position of the object within the spatial area.
  • This so-called determination position which indicates the position at which the value was determined within the spatial area at the corresponding object, is then used to determine the field with which the determined value is linked.
  • the above-described determination position is converted into a uniform position coordinate system or uniform world coordinates so that the position information of the determined values can be compared with one another. In this way, the values can be entered in the corresponding fields regardless of the sensor position.
  • step c) it is particularly advantageous if the detection of a situation change in step c) takes place as a function of the comparison between a learned behavior from step b) and a behavior based on the statistical evaluation of currently determined values.
  • a current behavior for one or more fields is continuously determined and compared with a previously learned behavior for the corresponding fields. This makes it possible to detect changes in the situation that affect the behavior of the objects within one or more fields, with respect to one or more object properties.
  • a situation change is detected as a function of the comparison between the learned behavior from step b) and current values of a trajectory of a specific object.
  • values relating to one or more object properties of a specific object are detected, while the object moves on a specific path (trajectory) through the area to be monitored. These values recorded in this way are then assigned to a specific trajectory of the object and compared with the learned behavior of the other objects. If the behavior with respect to one or more object properties of the specific object deviates significantly compared with the learned behavior of the other objects, it is possible to conclude that the situation has changed.
  • individual objects can be identified, which behave atypically compared to the typical behavior of the entire scene, which also represents a situation change.
  • mean values, sums, products and / or standard deviations are formed for the statistical evaluation, which can then be compared with one another. In principle, all arithmetic operations are possible which the expert would use for statistical purposes.
  • Particularly suitable sensors for monitoring the spatial area are imaging sensors which are advantageously equipped with image recognition methods, radar sensors or RFID transponders.
  • objects to be monitored come in particular people, animals, ie living beings in general, road, rail, air or water vehicles into consideration as well as particles and other objects, It is particularly advantageous if the sensors mentioned for determining properties such as activity of Objects that are set to speed the objects or their direction.
  • the threshold value can be designed such that sudden changes in the behavior of the objects are recognized as a change in situation.
  • threshold values can also be stored for a field if several typical behaviors arise with regard to an object property.
  • the invention relates to a computer program product for carrying out the above method and a device for this purpose.
  • Fig. 1a and 1b sketchy show the inclusion of a road junction, which is to be used as an embodiment of the further embodiments.
  • the intersection is monitored by two video cameras, where Fig. 1a the image of the first video camera can be seen from a first angle while Fig. 1b See the image of a second video camera from a different angle.
  • values of specific object properties are identified from the recorded image information.
  • the objects are vehicles passing through the intersection.
  • the speed and direction are recognized by the image recognition software for each vehicle and stored accordingly.
  • per vehicle which is in the field of view of the cameras, continuously determined both the speed and the direction.
  • the recorded image area is subdivided into a plurality of fields 1.
  • this is only sketchily indicated and usually extends over the entire scene.
  • the position at which the corresponding values were determined is also determined. So that these values can also be utilized independently of the location of the corresponding camera, this position information is converted into a world coordinate system and stored accordingly.
  • a vehicle which by the field of view of the two cameras in Fig. 1a and 1b a plurality of speed and direction values at different positions or a plurality of speed and direction values at the same positions, since the values can be detected by a plurality of different sensors.
  • Each speed and direction value is now assigned the corresponding field 1 within which the determined position lies, based on its position.
  • exactly one such field can be assigned to each measured speed and direction value.
  • Fig. 2 shows.
  • the hatched fields 2 are the fields in which object values of different objects have been stored. The white fields, however, have no stored values.
  • a derived card-like structure is in Fig. 3 to recognize.
  • Fig. 3 This is a data structure in which the mean value of the values stored in this field was formed in each field. In this embodiment, this was done with the object property "Speed". But it can also derive other card-like structures, such. For example, those in which the standard deviation is determined as a statistical evaluation for each field.
  • the values are determined for each field whose property is to be derived statistically. In this embodiment, these would be the speed values. Then the statistical evaluation is carried out for each field. Since the statistical evaluation is carried out separately for each field, this results in different fields for each field Fig. 3 represented by a different hatching in the individual fields.
  • the behavior of the objects can be determined with respect to certain object properties within the scene, whereby the learned behavior is understood as a typical behavior of the objects. Now changes the scene by z. If, for example, the speed becomes slower or faster, this ultimately also results in the statistical evaluation so that a situation change can then be concluded on the basis of a comparison between the learned behavior and the current values. Thus, such a situation change is ultimately always a deviation of the behavior of the objects from the typical behavior of the objects within this scene with respect to at least one object property.
  • Fig. 4 shows by way of example the comparison between the learned behavior and a current behavior with respect to the two object properties speed (S) and direction (D) on the basis of the statistical evaluation "averaging" (S ⁇ , D ⁇ ).
  • map-type data structures have been learned as described above. For this purpose, data structures with respect to the speed and direction of the objects within the observed scene were learned. From the determined values for the speed S and the direction D, the average value was then derived for each field, which is referred to as card-like data structure 41 and 42 in FIG Fig. 4 can be seen.
  • a common behavior was derived, which is also stored in the form of a card-like data structure 43 and 44 respectively.
  • a common learned behavior can be done with the help of different weights z. B. younger learned behavior to be considered more than older.
  • the system is self-learning and it adapts to the new situation.
  • the system automatically adapts to the given situation without having to readjust it. If the traffic jam clears up again after a while, the average speed increases abruptly, which is again recognized as a change in the situation, as is the behavior of the objects from the typical behavior learned differs.
  • the system can recognize both the formation of a congestion and its resolution afterwards without any problems and manual intervention and adapts by learning the standard deviation also to expected levels of change.
  • the average value (mean) in a large scene often varies only slightly, although e.g. has jammed on a roadway, e.g. because the traffic jam initially forms only on one lane and then "spreads" to the other lanes or the jam forms only in one direction, while the opposite lane is free. This can lead to compensatory changes that cancel each other out, so that no change in the situation is detected in the sum.
  • the standard deviation is stored for each field, which is then used for a threshold value comparison, so as to be able to determine selectively for individual fields situation changes.
  • a sensitivity can be set so that in some fields situation changes result, since here the standard deviations are relatively low, while in other fields this is recognized as normal behavior due to relatively large standard deviations.
  • the method of the invention adjusts the standard deviation in the fields where a large change occurs over time (eg, when a congestion occurs), while the other fields remain sensitive to changes (eg, the oncoming lane where no congestion has occurred ).
  • Fig. 5 schematically shows another embodiment in which a particular or specific object was tracked by means of a corresponding tracking algorithm.
  • a trajectory 51 is recorded by the object to be observed, which has a plurality of data points. At these data points were then z. B. corresponding values of object properties to be measured and stored. These data points are then compared to the learned behavior of the overall scene to determine if the object behaves in a typical or atypical manner relative to the overall behavior of all objects. This is done at the same time for all objects in the scene individually.
  • each data point of the trajectory is compared with the underlying card-like data structure, in particular with the fields in which the corresponding data point falls.
  • this is shown in a scene with the data point 52.
  • the corresponding object property here the speed
  • Fig. 5 can be with this embodiment in Fig. 5 recognize if z. B, Cyclists move very fast through a scene or pedestrians who run aimlessly or suddenly start to run or walk over otherwise barely used areas like meadows. Also can be seen with this system a ghost driver. But it can also monitor other objects such as aircraft so that when monitoring an airport z. For example, situations may be identified where service vehicles are moving in unusual areas or at atypical speeds. It is also possible to detect launches of rarely used runways as well as changes in the starting direction and the use of a test vehicle with this method.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)
EP10004126.8A 2009-05-18 2010-04-19 Procédé de reconnaissance automatique d'une modification de situation Not-in-force EP2254104B1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
DE200910021765 DE102009021765A1 (de) 2009-05-18 2009-05-18 Verfahren zum automatischen Erkennen einer Situationsänderung

Publications (3)

Publication Number Publication Date
EP2254104A2 true EP2254104A2 (fr) 2010-11-24
EP2254104A3 EP2254104A3 (fr) 2012-11-07
EP2254104B1 EP2254104B1 (fr) 2014-06-11

Family

ID=42262035

Family Applications (1)

Application Number Title Priority Date Filing Date
EP10004126.8A Not-in-force EP2254104B1 (fr) 2009-05-18 2010-04-19 Procédé de reconnaissance automatique d'une modification de situation

Country Status (2)

Country Link
EP (1) EP2254104B1 (fr)
DE (1) DE102009021765A1 (fr)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6587781B2 (en) 2000-08-28 2003-07-01 Estimotion, Inc. Method and system for modeling and processing vehicular traffic data and information and applying thereof
US7620402B2 (en) 2004-07-09 2009-11-17 Itis Uk Limited System and method for geographically locating a mobile device
DE102010053150B4 (de) * 2010-11-30 2018-02-01 Deutsches Zentrum für Luft- und Raumfahrt e.V. Verfahren und Vorrichtung zur Auswertung einer Verkehrslage
DE102021207997A1 (de) 2021-07-26 2023-01-26 Robert Bosch Gesellschaft mit beschränkter Haftung Verfahren zum Überprüfen einer Vollständigkeit eines Modells einer Verkehrsdynamik an einem Verkehrsknotenpunkt

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0798684A1 (fr) 1996-03-25 1997-10-01 MANNESMANN Aktiengesellschaft Procédé et système de saisie de situation de la circulation par un dispositif fixe de saisie de données
US20030107650A1 (en) 2001-12-11 2003-06-12 Koninklijke Philips Electronics N.V. Surveillance system with suspicious behavior detection
US20040130620A1 (en) 2002-11-12 2004-07-08 Buehler Christopher J. Method and system for tracking and behavioral monitoring of multiple objects moving through multiple fields-of-view

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5091780A (en) * 1990-05-09 1992-02-25 Carnegie-Mellon University A trainable security system emthod for the same
JPH06153202A (ja) * 1992-10-29 1994-05-31 F M T:Kk 異常監視装置
US5448484A (en) * 1992-11-03 1995-09-05 Bullock; Darcy M. Neural network-based vehicle detection system and method
DE4300651A1 (de) * 1993-01-08 1994-07-14 Refit Ev Verfahren zur Ermittlung von Verkehrsdaten
US20060074546A1 (en) * 1999-04-19 2006-04-06 Dekock Bruce W System for providing traffic information
JP3243234B2 (ja) * 1999-07-23 2002-01-07 松下電器産業株式会社 混雑度計測方法、計測装置、およびそれを用いたシステム
US6587781B2 (en) * 2000-08-28 2003-07-01 Estimotion, Inc. Method and system for modeling and processing vehicular traffic data and information and applying thereof
JP2005173668A (ja) * 2003-12-08 2005-06-30 Hitachi Ltd 生活行動パターンの異常判定システム及びそのための装置
JP4471362B2 (ja) * 2004-08-25 2010-06-02 パナソニック株式会社 監視カメラ装置
JP4368767B2 (ja) * 2004-09-08 2009-11-18 独立行政法人産業技術総合研究所 異常動作検出装置および異常動作検出方法
US8009193B2 (en) * 2006-06-05 2011-08-30 Fuji Xerox Co., Ltd. Unusual event detection via collaborative video mining

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0798684A1 (fr) 1996-03-25 1997-10-01 MANNESMANN Aktiengesellschaft Procédé et système de saisie de situation de la circulation par un dispositif fixe de saisie de données
US20030107650A1 (en) 2001-12-11 2003-06-12 Koninklijke Philips Electronics N.V. Surveillance system with suspicious behavior detection
US20040130620A1 (en) 2002-11-12 2004-07-08 Buehler Christopher J. Method and system for tracking and behavioral monitoring of multiple objects moving through multiple fields-of-view

Also Published As

Publication number Publication date
EP2254104B1 (fr) 2014-06-11
EP2254104A3 (fr) 2012-11-07
DE102009021765A1 (de) 2010-11-25

Similar Documents

Publication Publication Date Title
DE102014008578B4 (de) Verfahren zur Ermittlung von Positionsdaten zur Nutzung beim Betrieb eines Fahrzeugsystems eines Kraftfahrzeugs und Positionsdatenermittlungs- und-verteilssystem
DE102017203838B4 (de) Verfahren und System zur Umfelderfassung
EP2422332B1 (fr) Procede pour classer un objet comme un obstacle
DE102009033431B4 (de) Verfahren und Vorrichtung zur dynamischen Steuerung einer Signalanlage
DE102018221054B4 (de) Verfahren zum Bereitstellen von Kartendaten in einem Kraftfahrzeug, Kraftfahrzeug und zentrale Datenverarbeitungseinrichtung
DE102014009059B4 (de) Seiten-Rückwärts-Alarmsystem für Fahrzeuge und Alarmsteuerverfahren für dieses
DE102018218220A1 (de) Steuergerät für ein Fahrzeug
EP2867093B1 (fr) Procédé et système d'utilisation de l'information
DE102011010864A1 (de) Verfahren und System zur Vorhersage von Kollisionen
WO2013135535A1 (fr) Procédé pour la détection d'un embouteillage au moyen d'une communication sans fil voiture-à-voiture
EP1327969B1 (fr) Véhicule avec moyens pour la reconnaissance de panneaux de signalisation routière
DE102020003073B3 (de) Verfahren und Vorrichtung zum automatisierten Fahrbetrieb eines Fahrzeuges und Fahrzeug
WO2019162794A1 (fr) Procédé et système pour la détection de places de stationnement appropriées pour un véhicule
DE102021200636B4 (de) Verfahren zum Erzeugen einer Fahrspurwechselempfehlung, Spurwechselassistenzsystem sowie Kraftfahrzeug mit einem Spurwechselassistenzsystem
DE102021123270B3 (de) Verfahren und Steuerschaltung zum Überprüfen, ob ein aktuell aktiver Fahrmodus innerhalb seiner ODD betrieben wird, sowie System und Backend-Server
DE102017212644A1 (de) Sensorsystem und Verfahren zum Ermitteln wenigstens einer Verkehrssituation
EP2110798A1 (fr) Procédé de surveillance d'une voie de trafic pour un moyen de transport d'un type prédéterminé
DE102019211098B4 (de) Verfahren, Vorrichtung und Computerprogramm zum Bestimmen einer Ampelphase einer Ampel einer Ampelanlage
DE102017212513A1 (de) Verfahren und System zum Detektieren eines freien Bereiches innerhalb eines Parkplatzes
DE102022206348A1 (de) Verfahren und System zum Ermitteln von Sensor- und/oder Lokalisierungsfehlern
EP2254104B1 (fr) Procédé de reconnaissance automatique d'une modification de situation
EP3770879B1 (fr) Procédé, dispositif et programme informatique permettant de traiter des données au niveau d'un feu de signalisation
DE102020005763A1 (de) Verfahren zur Steuerung eines Fahrzeugs
DE102009037461A1 (de) Assistenzsystem zur Erhöhung einer Verkehrssicherheit
WO2023237424A1 (fr) Procédé et système d'assistance pour prédire un trajet de conduite, et véhicule à moteur

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA ME RS

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

AX Request for extension of the european patent

Extension state: AL BA ME RS

RIC1 Information provided on ipc code assigned before grant

Ipc: G08G 1/01 20060101ALI20121002BHEP

Ipc: G08G 1/04 20060101AFI20121002BHEP

17P Request for examination filed

Effective date: 20130507

RBV Designated contracting states (corrected)

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

INTG Intention to grant announced

Effective date: 20140128

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 672570

Country of ref document: AT

Kind code of ref document: T

Effective date: 20140715

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 502010007155

Country of ref document: DE

Effective date: 20140724

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140912

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140911

REG Reference to a national code

Ref country code: NL

Ref legal event code: VDEP

Effective date: 20140611

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20141013

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20141011

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502010007155

Country of ref document: DE

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

26N No opposition filed

Effective date: 20150312

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502010007155

Country of ref document: DE

Effective date: 20150312

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 502010007155

Country of ref document: DE

Representative=s name: GRAMM, LINS & PARTNER PATENT- UND RECHTSANWAEL, DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20150419

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20150419

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150419

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150430

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150430

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20151231

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150419

REG Reference to a national code

Ref country code: AT

Ref legal event code: MM01

Ref document number: 672570

Country of ref document: AT

Kind code of ref document: T

Effective date: 20150419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150419

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20100419

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20150430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20140611

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20200317

Year of fee payment: 11

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 502010007155

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20211103