EP1077013B1 - Procede et dispositif pour le fonctionnement de systemes a commande vocale installes dans des vehicules automobiles - Google Patents

Procede et dispositif pour le fonctionnement de systemes a commande vocale installes dans des vehicules automobiles Download PDF

Info

Publication number
EP1077013B1
EP1077013B1 EP99924872A EP99924872A EP1077013B1 EP 1077013 B1 EP1077013 B1 EP 1077013B1 EP 99924872 A EP99924872 A EP 99924872A EP 99924872 A EP99924872 A EP 99924872A EP 1077013 B1 EP1077013 B1 EP 1077013B1
Authority
EP
European Patent Office
Prior art keywords
voice
signal
microphone
loudspeaker
signals
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Lifetime
Application number
EP99924872A
Other languages
German (de)
English (en)
Other versions
EP1077013A1 (fr
Inventor
Klaus Schaaf
Jürgen Schultz
Volker THÖRMANN
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Volkswagen AG
Original Assignee
Volkswagen AG
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from DE19827134A external-priority patent/DE19827134A1/de
Application filed by Volkswagen AG filed Critical Volkswagen AG
Publication of EP1077013A1 publication Critical patent/EP1077013A1/fr
Application granted granted Critical
Publication of EP1077013B1 publication Critical patent/EP1077013B1/fr
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers
    • H04R3/02—Circuits for transducers for preventing acoustic reaction, i.e. acoustic oscillatory feedback

Definitions

  • the invention relates to a method and a device for operating voice-assisted Systems such as communication and / or speech / intercom systems in motor vehicles, in which or via which Mehdachmlkrofonan extract recorded voice signals and at least one Speakers are passed on, according to the preamble of claims 1 and 5.
  • EP 0078014 B1 is a speech recognition device for a motor vehicle is known, in which in the amplifier system of the speech recognition device Sensors are reported or fed in whether the engine is in operation and / or that Vehicle moves. Then a level control is then attempted will filter out the voice command from the background noise.
  • a filtering is known from WO 97/34290 in which periodic interference signals are filtered out, in which their period is determined and by means of a generator is interfered so that the speech signal remains.
  • DE 39 25 589 A1 discloses the use of a multiple microphone arrangement, being one of the microphones in the engine compartment and a when used in a motor vehicle is further arranged in the passenger compartment. The two signals are then subtracted. The result is that only the engine noise or the actual one Operating noise of the vehicle itself subtracted from the overall signal in the passenger compartment becomes. Specific background noises are not taken into account here. As well there is no feedback suppression, which is a particular problem. All over where microphones and loudspeakers are arranged in acoustically connectable proximity, it happens that the acoustic signal coupled out at the loudspeaker in turn the microphone feeds back. There is a so-called feedback and one subsequent override.
  • the invention is therefore based on the object of a method and a device of generic type to the extent that feedback and Instabilities that occur when several microphones and loudspeakers are arranged, be suppressed.
  • the object is in a method of the generic type according to the invention solved by the characterizing features of claim 1.
  • the invention is based on both the method and the device Communication and / or intercom / intercom in motor vehicles. It it is also known to arrange a multiple microphone arrangement, in addition Record voice and noise signals and the total signal Subtract noise signals again so that the speech signal remains filtered.
  • the essence of the invention is that the respective Microphone signal is first frequency shifted by a small amount ⁇ F, and only then on the speaker or on the input of a voice-controlled Facility is given.
  • the frequency shift according to the invention is carried out at a defined point and is not arbitrary, supports the on the one hand the filtering and on the other hand there are feedbacks, also the echo signal decoupled by one of the not yet frequency shifted overall signal first microphone the overall signal of another or second shifted by ⁇ F Microphones is subtracted, and vice versa.
  • the illustration shows the basic structure as well as the mode of operation, so that the image itself, both the procedural measures and the interconnection of the individual furnishing elements to each other in their logical entirety is or are recognizable.
  • the vehicle interior is in divided into two sub-rooms, namely front and rear.
  • the microphone M1 picks up the speech signal there and possibly noise signals.
  • the noise signal consists of the noise level in the passenger compartment that results from the operation of the vehicle. This can be engine noises, wind noises and rolling noises, but also acoustic echo signals from the other tell room and the like.
  • the sum signal from speech and noise backdrop (total signal) contained at M1 is fed to a first summation point S1.
  • a correspondingly prepared signal is then also generated from this summation point an acoustic model AM1 in front, fed in the acoustic model AM1 generated subtraction signal in this embodiment comes from the im received rear part of the vehicle, and already frequency-shifted signal. Because this signal coming from M2 and frequency shifted in F2, which originates in the rear compartment of the passenger cell, via AM1 signaling is also taken into account at the front, that generated in the rear compartment of the vehicle and after vome, acoustically transported into the front compartment of the passenger cell Signal, which is also registered by M1, in turn at the summation point S1 subtracted. That is, the device AM1 becomes the rear compartment of the passenger compartment acoustically separated from the front compartment of the passenger compartment.
  • the total perceptible acoustic signal is fed in, and at the summation point S1 first subtracts the echo from the rear compartment of the passenger compartment.
  • the so obtained original signal from M1 from the front compartment of the passenger cell then becomes one Frequency shifting device F.1 supplied and by an amount ⁇ F, for example 5 Hz, shifted.
  • the output signal of F1 thus obtained is then the Loudspeaker L1 of the rear compartment of the passenger compartment supplied and to the other at the same time again fed into the device AM2 in the same way.
  • AM2 again represents the acoustic model for the rear part of the Passenger compartment.
  • the transmission of a voice message from the back of the tell room Passenger cell via M2 to the front compartment of the passenger cell via L2 takes place in analogous way. That is, the microphone M2 registers the voice message together Noise in the rear compartment of the passenger compartment and transmits it to the Summation point S2, at which the acoustic recorded via M1 Overall signal, i.e. the echo and background noise is subtracted. So again the echo-free signal created by the microphone M2 then also becomes one Frequency shifter F2 supplied, which in turn a frequency shift by an amount ⁇ F. At the output of this frequency shifter F2 becomes the result or the signal thus processed in turn the front subspace the passenger compartment, namely the loudspeaker L2 positioned there.
  • the Frequency shift for transmission from front to back can also be from that frequency shift from the back to the front be different.

Landscapes

  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Soundproofing, Sound Blocking, And Sound Damping (AREA)
  • Interconnected Communication Systems, Intercoms, And Interphones (AREA)

Claims (7)

  1. Procédé destiné à l'utilisation de systèmes à commande vocale, tels que des dispositifs de communication et/ou de conversation en duplex, embarqués dans des véhicules automobiles, dans lequel des signaux vocaux et des signaux de bruit de fond sont captés sous forme de signaux globaux par un système à plusieurs microphones (M1, M2) et sont transmis vers au moins un haut-parleur (L1, L2), le signal global capté par chaque microphone (M1, M2) étant d'abord soumis à un décalage de la fréquence d'une valeur Δ F et ensuite seulement transmis vers le ou les haut-parleurs (L1, L2) ou vers l'entrée d'un dispositif à commande vocale,
       caractérisé en ce que, pour éliminer les rétroactions et les signaux d'écho, l'on soustrait du signal global, n'ayant pas encore subi de décalage de fréquence, d'un premier microphone (M1, M2), le signal décalé de Δ F d'un autre ou deuxième microphone (M2, M1), et inversement.
  2. Procédé destiné à l'utilisation de systèmes à commande vocale selon la revendication 1,
       caractérisé en ce que, pour le couplage additif ou la soustraction acoustique des signaux de bruit de fond, un modèle acoustique (AM1, AM2) quelconque est formé à partir des signaux globaux enregistrés et est acheminé selon la technique de traitement de signaux, entre le microphone (M1, M2) correspondant et le dispositif de décalage de la fréquence (F1, F2) correspondant, vers un point de sommation (S1, S2) correspondant en vue de la soustraction.
  3. Procédé destiné à l'utilisation de systèmes à commande vocale selon la revendication 2,
       caractérisé en ce que l'habitacle du véhicule est divisé en au moins deux espaces acoustiques partiels, de telle sorte que chaque partie comporte au moins un site pour microphones et aussi au moins un site pour haut-parleurs, en ce que ledit décalage de la fréquence Δ F est effectué entre le site du microphone de l'une des parties et le site du haut-parleur de l'autre partie et lesdits modèles acoustiques (AM1, AM2) sont appliqués entre les sites pour haut-parleurs et les sites pour microphones de l'une des parties et entre les sites pour haut-parleurs et les sites pour microphones de l'autre partie, de telle sorte qu'il se forme un circuit de régulation électroacoustique fermé au sens de la technique des signaux.
  4. Procédé destiné à l'utilisation de systèmes à commande vocale selon la revendication 3,
       caractérisé en ce que, par l'intermédiaire desdits modèles acoustiques (AM1, AM2), sont pris en compte non seulement les signaux vocaux et/ou les signaux de bruit de fond des différentes parties de l'habitacle du véhicule, mais en plus, sont aussi pris en compte les bruits générés dans l'ensemble de l'environnement et sont soustraits du signal global, de telle sorte qu'il ne reste essentiellement que le signal vocal.
  5. Dispositif destiné à l'utilisation de systèmes à commande vocale, tels que des dispositifs de communication et/ou de conversation en duplex, embarqués dans des véhicules automobiles, comprenant une pluralité de microphones (M1, M2) et de haut-parleurs (L1, L2), ainsi que des moyens destinés à transmettre des informations vocales ou des instructions vocales, sachant que l'habitacle du véhicule est divisé en au moins deux parties, éventuellement ouvertes, comprenant chacune au moins un microphone (M1, M2) et au moins un haut-parleur (L1, L2), et lesdits moyens comportent également des dispositifs de décalage de la fréquence (F1, F2), qui sont montés respectivement entre l'un des microphones (M1, M2) et le haut-parleur (L1, L2) correspondant situé dans l'autre partie,
       caractérisé en ce que chaque signal du haut-parleur qui en résulte est capté parallèlement et superposé par soustraction dans un point de sommation (S1, S2) respectif au signal global du microphone installé dans la même partie, et les éléments de l'ensemble du dispositif sont câblés pour former un circuit de régulation électroacoustique fermé.
  6. Dispositif destiné à l'utilisation de systèmes à commande vocale selon la revendication 5,
       caractérisé en ce que, entre la détection parallèle de chaque signal du haut-parleur et le point de sommation (S1, S2) correspondant, il est prévu des moyens (AM1, AM2), dans lesquels sont générés des modèles dits acoustiques, qui influencent et/ou traitent chaque signal du haut-parleur, et en ce que le signal résultant de ces moyens (AM1, AM2) est transmis vers le point de sommation (S1, S2) correspondant.
  7. Dispositif destiné à l'utilisation de systèmes à commande vocale selon la revendication 6,
       caractérisé en ce que les modèles acoustiques (AM1, AM2) contiennent des moyens pour la reconnaissance des modèles de bruits, qui sont destinés à assurer une séparation entre bruits du moteur et/ou bruits de roulement et signaux acoustiques générés par la voix, ainsi qu'à assurer la séparation entre signaux vocaux primaires et signaux de l'écho renvoyés.
EP99924872A 1998-05-06 1999-05-04 Procede et dispositif pour le fonctionnement de systemes a commande vocale installes dans des vehicules automobiles Expired - Lifetime EP1077013B1 (fr)

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
DE19820000 1998-05-06
DE19820000 1998-05-06
DE19827134 1998-06-18
DE19827134A DE19827134A1 (de) 1998-05-06 1998-06-18 Verfahren und Einrichtung zum Betrieb von sprachunterstützten Systemen in Kraftfahrzeugen
PCT/EP1999/003031 WO1999057938A1 (fr) 1998-05-06 1999-05-04 Procede et dispositif pour le fonctionnement de systemes a commande vocale installes dans des vehicules automobiles

Publications (2)

Publication Number Publication Date
EP1077013A1 EP1077013A1 (fr) 2001-02-21
EP1077013B1 true EP1077013B1 (fr) 2002-03-06

Family

ID=26045962

Family Applications (1)

Application Number Title Priority Date Filing Date
EP99924872A Expired - Lifetime EP1077013B1 (fr) 1998-05-06 1999-05-04 Procede et dispositif pour le fonctionnement de systemes a commande vocale installes dans des vehicules automobiles

Country Status (5)

Country Link
US (1) US7010129B1 (fr)
EP (1) EP1077013B1 (fr)
JP (1) JP2002514775A (fr)
ES (1) ES2174611T3 (fr)
WO (1) WO1999057938A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1445761A1 (fr) 2003-02-07 2004-08-11 Volkswagen Aktiengesellschaft Appareil et méthode pour le fonctionnement de systèmes assistés par la parole dans des véhicules automobiles
WO2005018277A1 (fr) 2003-07-18 2005-02-24 Volkswagen Aktiengesellschaft Dispositif et procede pour exploiter des systemes a assistance vocale dans des automobiles
EP1625973A1 (fr) 2004-08-10 2006-02-15 Volkswagen Aktiengesellschaft Système de support vocal pour véhicule automobile

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3886391B2 (ja) * 2002-02-15 2007-02-28 シャープ株式会社 カード型装置、及び、それを備えた電子機器
US8537030B2 (en) * 2010-02-15 2013-09-17 Ford Global Technologies, Llc Pedestrian alert system and method
US9628930B2 (en) 2010-04-08 2017-04-18 City University Of Hong Kong Audio spatial effect enhancement
JP6111955B2 (ja) * 2013-09-27 2017-04-12 アンデン株式会社 車両接近通報装置
JP6231042B2 (ja) * 2015-03-24 2017-11-15 本田技研工業株式会社 能動騒音抑制システム、及び車両用能動騒音抑制システム
US9595253B2 (en) 2015-03-24 2017-03-14 Honda Motor Co., Ltd. Active noise reduction system, and vehicular active noise reduction system

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS5870292A (ja) 1981-10-22 1983-04-26 日産自動車株式会社 車両用音声認識装置
JPH0746791B2 (ja) 1985-07-17 1995-05-17 日本電信電話株式会社 ハウリング抑圧回路
GB2208990B (en) * 1987-08-19 1991-04-03 Thomas Mcgregor Voice enhancer system
US4912767A (en) 1988-03-14 1990-03-27 International Business Machines Corporation Distributed noise cancellation system
DE3925589C2 (de) * 1989-08-02 1994-03-17 Blaupunkt Werke Gmbh Verfahren und Anordnung zur Störbefreiung von Sprachsignalen
DE4106405C2 (de) 1990-03-23 1996-02-29 Ricoh Kk Geräuschunterdrückungseinrichtung für ein Spracherkennungsystem
GB9026906D0 (en) * 1990-12-11 1991-01-30 B & W Loudspeakers Compensating filters
DE4203436A1 (de) 1991-02-06 1992-08-13 Koenig Florian Datenreduzierte sprachkommunikation
DE4227826C2 (de) 1991-08-23 1999-07-22 Hitachi Ltd Digitales Verarbeitungsgerät für akustische Signale
JP2760373B2 (ja) * 1995-03-03 1998-05-28 日本電気株式会社 雑音消去装置
DE19524847C1 (de) 1995-07-07 1997-02-13 Siemens Ag Vorrichtung zur Verbesserung gestörter Sprachsignale
US5903819A (en) 1996-03-13 1999-05-11 Ericsson Inc. Noise suppressor circuit and associated method for suppressing periodic interference component portions of a communication signal
US5920834A (en) * 1997-01-31 1999-07-06 Qualcomm Incorporated Echo canceller with talk state determination to control speech processor functional elements in a digital telephone system
DE19705471C2 (de) 1997-02-13 1998-04-09 Sican F & E Gmbh Sibet Verfahren und Schaltungsanordnung zur Spracherkennung und zur Sprachsteuerung von Vorrichtungen
US5893056A (en) * 1997-04-17 1999-04-06 Northern Telecom Limited Methods and apparatus for generating noise signals from speech signals
US6098040A (en) * 1997-11-07 2000-08-01 Nortel Networks Corporation Method and apparatus for providing an improved feature set in speech recognition by performing noise cancellation and background masking

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1445761A1 (fr) 2003-02-07 2004-08-11 Volkswagen Aktiengesellschaft Appareil et méthode pour le fonctionnement de systèmes assistés par la parole dans des véhicules automobiles
WO2005018277A1 (fr) 2003-07-18 2005-02-24 Volkswagen Aktiengesellschaft Dispositif et procede pour exploiter des systemes a assistance vocale dans des automobiles
EP1625973A1 (fr) 2004-08-10 2006-02-15 Volkswagen Aktiengesellschaft Système de support vocal pour véhicule automobile

Also Published As

Publication number Publication date
JP2002514775A (ja) 2002-05-21
EP1077013A1 (fr) 2001-02-21
US7010129B1 (en) 2006-03-07
WO1999057938A1 (fr) 1999-11-11
ES2174611T3 (es) 2002-11-01

Similar Documents

Publication Publication Date Title
DE3925589C2 (de) Verfahren und Anordnung zur Störbefreiung von Sprachsignalen
EP2296356B1 (fr) Procédé et dispositif d'analyse et de détermination de propriétés acoustiques d'un dispositif téléphonique mains libres de véhicule automobile
DE4021969C2 (de) Automatische Lautstärkeregelung
DE69318926T2 (de) Tonwiedergabesystem
DE69427323T2 (de) Akustisches Wiedergabesystem
DE19938158C1 (de) Verfahren und Vorrichtung sowie ihre Verwendung zur Kompensation von Verlusten eines akustischen Signals
DE19938171C2 (de) Verfahren zur Verarbeitung von akustischen Signalen und Kommunikationsanlage für Insassen in einem Fahrzeug
DE60031583T2 (de) Sprachverteilungssystem
EP3375204B1 (fr) Traitement de signal audio dans un véhicule
EP1077013B1 (fr) Procede et dispositif pour le fonctionnement de systemes a commande vocale installes dans des vehicules automobiles
DE10357800B3 (de) Hörgerät mit Störgeräuschunterdrückung und entsprechendes Störgeräuschunterdrückungsverfahren
EP1189419A2 (fr) Procede et appareil pour eliminer l'interference d'un haut-parleur sur de signaux de microphone
EP2200341B1 (fr) Procédé de fonctionnement d'un appareil d'aide auditive et appareil d'aide auditive doté d'un dispositif de séparation de sources
DE10025496C2 (de) Audiosystem, insbesondere für Kraftfahrzeuge
DE102006047983A1 (de) Verarbeitung eines Eingangssignals in einem Hörgerät
EP1445761B1 (fr) Appareil et méthode pour le fonctionnement de systèmes assistés par la parole dans des véhicules automobiles
DE19827134A1 (de) Verfahren und Einrichtung zum Betrieb von sprachunterstützten Systemen in Kraftfahrzeugen
EP1649719B1 (fr) Dispositif et procédé pour exploiter des systèmes à assistance vocale dans des automobiles
DE4430931A1 (de) Einrichtung zur fahrgeräuschabhängigen Steuerung der Lautstärke eines Autoradios
DE10140523B4 (de) Vorrichtung zum Feedback Cancelling bei der Ausgabe von Mikrofonsignalen durch Lautsprecher
DE3411494A1 (de) Verfahren zur unterdrueckung von schall sowie schaltungsanordnung und schallschutzvorrichtung zur durchfuehrung des verfahrens
EP1929465A1 (fr) Procede de reduction active du bruit et dispositif pour la mise en oeuvre dudit procede
EP1282297A2 (fr) Procede et dispositif pour la suppression du bruit et d'echo
DE102004039066B4 (de) Audiosystem für ein Kraftfahrzeug
DE102024121171A1 (de) Mikrofon-Anordnung für ein Kraftfahrzeug

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20001206

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): DE ES FR GB IT SE

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAG Despatch of communication of intention to grant

Free format text: ORIGINAL CODE: EPIDOS AGRA

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

17Q First examination report despatched

Effective date: 20010725

GRAH Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOS IGRA

REG Reference to a national code

Ref country code: GB

Ref legal event code: IF02

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE ES FR GB IT SE

REF Corresponds to:

Ref document number: 59900941

Country of ref document: DE

Date of ref document: 20020411

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20020606

ET Fr: translation filed
REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2174611

Country of ref document: ES

Kind code of ref document: T3

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20021209

REG Reference to a national code

Ref country code: DE

Ref legal event code: R084

Ref document number: 59900941

Country of ref document: DE

Effective date: 20120121

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 17

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20150529

Year of fee payment: 17

Ref country code: ES

Payment date: 20150622

Year of fee payment: 17

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20150529

Year of fee payment: 17

Ref country code: IT

Payment date: 20150529

Year of fee payment: 17

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20160504

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160504

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20170131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160531

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160504

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160505

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20180531

Year of fee payment: 20

REG Reference to a national code

Ref country code: ES

Ref legal event code: FD2A

Effective date: 20181128

REG Reference to a national code

Ref country code: DE

Ref legal event code: R071

Ref document number: 59900941

Country of ref document: DE

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230523