EP1431962A2 - Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen - Google Patents

Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen Download PDF

Info

Publication number
EP1431962A2
EP1431962A2 EP04100553A EP04100553A EP1431962A2 EP 1431962 A2 EP1431962 A2 EP 1431962A2 EP 04100553 A EP04100553 A EP 04100553A EP 04100553 A EP04100553 A EP 04100553A EP 1431962 A2 EP1431962 A2 EP 1431962A2
Authority
EP
European Patent Office
Prior art keywords
speech
highband
lowband
khz
encoder
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP04100553A
Other languages
English (en)
French (fr)
Other versions
EP1431962B1 (de
EP1431962A3 (de
Inventor
Alan V Mccree
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Texas Instruments Inc
Original Assignee
Texas Instruments Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Texas Instruments Inc filed Critical Texas Instruments Inc
Priority claimed from EP01000172A external-priority patent/EP1158495B1/de
Publication of EP1431962A2 publication Critical patent/EP1431962A2/de
Publication of EP1431962A3 publication Critical patent/EP1431962A3/de
Application granted granted Critical
Publication of EP1431962B1 publication Critical patent/EP1431962B1/de
Anticipated expiration legal-status Critical
Expired - Lifetime legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/02—Speech enhancement, e.g. noise reduction or echo cancellation
    • G10L21/038—Speech enhancement, e.g. noise reduction or echo cancellation using band spreading techniques
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/02—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders
    • G10L19/0204—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using spectral analysis, e.g. transform vocoders or subband vocoders using subband decomposition
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L19/00—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis
    • G10L19/04—Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis using predictive techniques
    • G10L19/08—Determination or coding of the excitation function; Determination or coding of the long-term prediction parameters
    • G10L19/12—Determination or coding of the excitation function; Determination or coding of the long-term prediction parameters the excitation function being a code excitation, e.g. in code excited linear prediction [CELP] vocoders

Definitions

  • the present invention relates to electronic devices, and, more particularly, to speech coding, transmission, storage, and decoding/synthesis methods and systems.
  • the performance of digital speech systems using low bit rates has become increasingly important with current and foreseeable digital communications.
  • Both dedicated channel and packetized-over-network (VoIP) transmission benefit from compression of speech signals.
  • the widely-used linear prediction (LP) digital speech coding compression method models the vocal tract as a time-varying filter and a time-varying excitation of the filter to mimic human speech.
  • M the order of the linear prediction filter, is taken to be about 10-12; the sampling rate to form the samples s(n) is typically taken to be 8 kHz (the same as the public switched telephone network (PSTN) sampling for digital transmission); and the number of samples ⁇ s(n) ⁇ in a frame is often 80 or 160 (10 or 20 ms frames).
  • Various windowing operations may be applied to the samples of the input speech frame.
  • ⁇ r(n) 2 yields the ⁇ a(j) ⁇ which furnish the best linear prediction.
  • the coefficients ⁇ a(j) ⁇ may be converted to line spectral frequencies (LSFs) for quantization and transmission or storage.
  • the ⁇ r(n) ⁇ form the LP residual for the frame, and ideally LP residual would be the excitation for the synthesis filter 1/A(z) where A(z) is the transfer function of equation (1).
  • the LP residual is not available at the decoder; thus the task of the encoder is to represent the LP residual so that the decoder can generate an LP excitation from the encoded parameters.
  • the LP compression approach basically only transmits/stores updates for the (quantized) filter coefficients, the (quantized) residual (waveform or parameters such as pitch), and the (quantized) gain.
  • a receiver regenerates the speech with the same perceptual characteristics as the input speech.
  • Figure 9 shows the blocks in an LP system. Periodic updating of the quantized items requires fewer bits than direct representation of the speech signal, so a reasonable LP coder can operate at bits rates as low as 2-3 kb/s (kilobits per second).
  • the ITU standard G.729 Annex E with a bit rate of 11.8 kb/s uses LP analysis with codebook excitation (CELP) to compress voiceband speech and has performance comparable to the 64 kb/s PCM used for PSTN digital transmission.
  • CELP codebook excitation
  • Another approach uses split-band CELP or MPLPC by coding a 4-8 kHz highband separately from the 0-4 kHz lowband and with fewer bits allocated to the highband; see Drogo de Jacovo et al, Some Experiments of 7 kHz Audio Coding at 16 kbit/s, IEEE ICASSP 1989, pp.192-195. Similarly, Tucker, Low Bit-Rate Frequency Extension Coding, IEE Colloquium on Audio and Music Technology 1998, pp.3/1-3/5, provides standard coding of the lowband 0-4 kHz plus codes the 4-8 kHz highband speech only for unvoiced frames (as determined in the lowband) and uses an LP filter of order 2-4 with noise excitation.
  • these approaches suffer from either too high a bit rate or too low a quality.
  • the present invention provides a method of speech coding, comprising: (a) partitioning a frame of digital speech into a lowband and a highband; (b) decimating the sampling rate of both said lowband and said highband; (c) encoding said decimated lowband from step (b) including a first method of quantization; (d) reversing the spectrum of a baseband image of said decimated highband from step (b); and (e) encoding the results of step (d) including said first method of quantization.
  • a method of decoding speech comprising: (a) decoding a first portion of an input signal as a lowband speech signal including using a first codebook; (b) decoding a second portion of an input signal as a highband speech signal including using said first codebook; and (c) combining the results of foregoing steps (a) and (b) to form a decoded wideband speech signal.
  • a wideband speech encoder comprising: (a) a lowband filter and a highband filter for digital speech; (b) a first encoder with input from said lowband filter; said first encoder using a first quantizer; (c) a second encoder with input from said highband filter, said second encoder using said first quantizer; and (d) a combiner for said first encoder and said second encoder to output encoded wideband speech.
  • a wideband speech decoder comprising: (a) a first speech decoder with an input for encoded narrowband speech and an LP codebook; (b) a second speech decoder with an input for encoded highband speech, said second decoder using said LP codebook.
  • the preferred embodiment systems include preferred embodiment encoders and decoders that process a wideband speech frame as the sum of a lowband signal and a highband signal in which the lowband signal has standalone speech encoding/decoding and the highband signal has encoding/decoding incorporating information from the lowband signal to modulate a noise excitation. This allows for a minimal number of bits to sufficiently encode the highband and yields an embedded coder.
  • Figure 1a shows in functional block format a first preferred embodiment system for wideband speech encoding, transmission (storage), and decoding including first preferred embodiment encoders and decoders.
  • the encoders and decoders use CELP lowband encoding and decoding plus a highband encoding and decoding incorporating information from the (decoded) lowband for modulation of a noise excitation with LP coding.
  • first preferred embodiment encoders proceed as follows.
  • the baseband of the decimated highband has a reversed spectrum because the baseband is an aliased image; see Figure 3b.
  • encode the first baseband (decimated lowband) signal with a (standard) narrowband speech coder.
  • the ITU G.729 standard 8 kb/s uses 18 bits for quantized LP coefficients (three codebooks) per 10 ms (80 samples) frame, 14 bits for pitch delay (adaptive codebook), 34 bits for delayed excitation differential (fixed codebook), and 14 bits for gains.
  • Figures 4a-4b show block diagrams of the encoder and decoder.
  • G.729 Annex E provides higher quality with a higher bit rate (11.8 kb/s).
  • Decoding reverses the encoding process by separating the highband and lowband code, using information from the decoded lowband to help decode the highband, and adding the decoded highband to the decoded lowband speech to synthesize wideband speech. See Figure 1c.
  • This split-band approach allows most of the code bits to be allocated to the lowband; for example, the lowband may consume 11.8 kb/s and the highband may add 2.2 kb/s for a total of 14 kb/s.
  • Figures 2a-2b illustrate the typical magnitudes of voiced and unvoiced speech, respectively, as functions of frequency over the range 0-8 kHz.
  • the bulk of the energy in voiced speech resides in the 0-3 kHz band.
  • the pitch structure (the fundamental frequency is about 125 Hz in Figure 2a) clearly appears in the range 0-3.5 kHz and persists (although jumbled) at higher frequencies.
  • the perceptual critical bandwidth at higher frequencies is roughly 10% of a band center frequency, so the individual pitch harmonics become indistinguishable and should require fewer bits for inclusion in a highband code.
  • Figure 2b shows unvoiced-speech energy peaks in the 3.5-6.5 kHz band.
  • the precise character of this highband signal contains little perceptual information.
  • the higher band (above 4 kHz) should require fewer bits to encode than the lower band (0-4 kHz).
  • This underlies the preferred embodiment methods of partitioning wideband (0-8 kHz) speech into a lowband (0-4 kHz) and a highband (4-8 kHz), recognizing that the lowband may be encoded by any convenient narrowband coder, and separately coding the highband with a relatively small number of bits as described in the following sections.
  • Figure 1b illustrates the flow of a first preferred embodiment speech coder which encodes at 14 kb/s with the following steps.
  • a first preferred embodiment decoding method essentially reverses the encoding steps for a bitstream encoded by the first preferred embodiment method.
  • a coded frame in the bitstream For a coded frame in the bitstream:
  • FIGS 8-9 show in functional block form preferred embodiment systems that use the preferred embodiment encoding and decoding.
  • the encoding and decoding can be performed with digital signal processors (DSPs) or general purpose programmable processors or application specific circuitry or systems on a chip such as both a DSP and RISC processor on the same chip with the RISC processor controlling.
  • Codebooks would be stored in memory at both the encoder and decoder, and a stored program in an onboard ROM or external flash EEPROM for a DSP or programmable processor could perform the signal processing.
  • Analog-to-digital converters and digital-to-analog converters provide coupling to the real world, and modulators and demodulators (plus antennas for air interfaces) provide coupling for transmission waveforms.
  • the encoded speech can be packetized and transmitted over networks such as the Internet.
  • Second preferred embodiment coders and decoders follow the first preferred embodiment coders and decoders and partition the sampled input into a lowband and a highband, downsample, and apply a narrowband coder to the lowband.
  • the second preferred embodiments vary the encoding of the highband with modulated noise-excited LP by deriving the modulation from the envelope of lbdh(m) rather than its absolute value. In particular, find the envelope en(m) of lbdh(m) by lowpass (0-1 kHz) filtering the absolute value
  • Figure 7 illustrates en(m) for the voiced speech of Figure 6 in the time domain.
  • the preferred embodiments may be modified in various ways while retaining the features of separately coding a lowband from a wideband signal and using information from the lowband to help encode the highband (remainder of the wideband) and/or using spectrum reversal for decimated highband LP coefficient quantization in order to obtain efficiency comparable to that for the lowband LP coefficient quantization.
  • the upper (2.8-3.8 kHz) portion of the lowband (0-4 kHz) could be replaced by some other portion(s) of the lowband for use as a modulation for the highband excitation.
  • the highband encoder/decoder may have its own LP analysis and quantization, so the spectral reversal would not be required; the wideband may be partitioned into a lowband plus two or more highbands; the lowband coder could be a parametric or even non-LP coder and a highband coder could be a waveform coder; and so forth.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Quality & Reliability (AREA)
  • Signal Processing (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Multimedia (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)
EP04100553A 2000-05-22 2001-05-22 Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen Expired - Lifetime EP1431962B1 (de)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US20615600P 2000-05-22 2000-05-22
US206156P 2000-05-22
EP01000172A EP1158495B1 (de) 2000-05-22 2001-05-22 Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen

Related Parent Applications (2)

Application Number Title Priority Date Filing Date
EP01000172A Division EP1158495B1 (de) 2000-05-22 2001-05-22 Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen
EP01000172A Division-Into EP1158495B1 (de) 2000-05-22 2001-05-22 Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen

Publications (3)

Publication Number Publication Date
EP1431962A2 true EP1431962A2 (de) 2004-06-23
EP1431962A3 EP1431962A3 (de) 2004-12-01
EP1431962B1 EP1431962B1 (de) 2006-04-05

Family

ID=32395343

Family Applications (1)

Application Number Title Priority Date Filing Date
EP04100553A Expired - Lifetime EP1431962B1 (de) 2000-05-22 2001-05-22 Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen

Country Status (1)

Country Link
EP (1) EP1431962B1 (de)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7783480B2 (en) 2004-09-17 2010-08-24 Panasonic Corporation Audio encoding apparatus, audio decoding apparatus, communication apparatus and audio encoding method
US7848921B2 (en) 2004-08-31 2010-12-07 Panasonic Corporation Low-frequency-band component and high-frequency-band audio encoding/decoding apparatus, and communication apparatus thereof
US7848925B2 (en) 2004-09-17 2010-12-07 Panasonic Corporation Scalable encoding apparatus, scalable decoding apparatus, scalable encoding method, scalable decoding method, communication terminal apparatus, and base station apparatus
RU2432624C1 (ru) * 2010-04-21 2011-10-27 Государственное образовательное учреждение высшего профессионального образования Академия Федеральной службы охраны Российской Федерации (Академия ФСО России) Способ уменьшения объема данных при широкополосном кодировании речевого сигнала
RU2455710C2 (ru) * 2008-01-31 2012-07-10 Фраунхофер-Гезелльшафт цур Фердерунг дер ангевандтен Устройство и способ расширения полосы пропускания аудио сигнала
US8301281B2 (en) 2006-12-25 2012-10-30 Kyushu Institute Of Technology High-frequency signal interpolation apparatus and high-frequency signal interpolation method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
ADIL BENYASSINE ET AL INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS: "MULTIBAND CELP CODING OF SPEECH" PROCEEDINGS OF THE ASILOMAR CONFERENCE ON SIGNALS, SYSTEMS AND COMPUTERS. PüACIFIC GROVE, NOV. 5 - 7, 1990, NEW YORK, IEEE, US, vol. VOL. 2 CONF. 24, 5 November 1990 (1990-11-05), pages 644-648, XP000280093 *
MCCREE A ED - INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS: "A 14 kb/s wideband speech coder with a parametric highband model" 2000 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING. PROCEEDINGS. (ICASSP). ISTANBUL, TURKEY, JUNE 5-9, 2000, IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING (ICASSP), NEW YORK, NY : IEEE, US, vol. VOL. 2 OF 6, 5 June 2000 (2000-06-05), pages II1153-II1156, XP002162018 ISBN: 0-7803-6294-2 *
TIAN W ET AL: "LOW-DELAY SUBBAND CELP CODING FOR WIDEBAND SPEECH" 1996 IEEE TENCON. DIGITAL SIGNAL PROCESSING APPLICATIONS PROCEEDINGS. PERTH, NOV. 26 - 29, 1996, IEEETENCON - DIGITAL SIGNAL PROCESSING APPLICATIONS, NEW YORK, NY : IEEE, US, vol. VOL. 1, 26 November 1996 (1996-11-26), pages 189-194, XP000781677 ISBN: 0-7803-3680-1 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7848921B2 (en) 2004-08-31 2010-12-07 Panasonic Corporation Low-frequency-band component and high-frequency-band audio encoding/decoding apparatus, and communication apparatus thereof
US7783480B2 (en) 2004-09-17 2010-08-24 Panasonic Corporation Audio encoding apparatus, audio decoding apparatus, communication apparatus and audio encoding method
US7848925B2 (en) 2004-09-17 2010-12-07 Panasonic Corporation Scalable encoding apparatus, scalable decoding apparatus, scalable encoding method, scalable decoding method, communication terminal apparatus, and base station apparatus
US8712767B2 (en) 2004-09-17 2014-04-29 Panasonic Corporation Scalable encoding apparatus, scalable decoding apparatus, scalable encoding method, scalable decoding method, communication terminal apparatus, and base station apparatus
US8301281B2 (en) 2006-12-25 2012-10-30 Kyushu Institute Of Technology High-frequency signal interpolation apparatus and high-frequency signal interpolation method
RU2455710C2 (ru) * 2008-01-31 2012-07-10 Фраунхофер-Гезелльшафт цур Фердерунг дер ангевандтен Устройство и способ расширения полосы пропускания аудио сигнала
RU2432624C1 (ru) * 2010-04-21 2011-10-27 Государственное образовательное учреждение высшего профессионального образования Академия Федеральной службы охраны Российской Федерации (Академия ФСО России) Способ уменьшения объема данных при широкополосном кодировании речевого сигнала

Also Published As

Publication number Publication date
EP1431962B1 (de) 2006-04-05
EP1431962A3 (de) 2004-12-01

Similar Documents

Publication Publication Date Title
US7330814B2 (en) Wideband speech coding with modulated noise highband excitation system and method
US7136810B2 (en) Wideband speech coding system and method
JP4662673B2 (ja) 広帯域音声及びオーディオ信号復号器における利得平滑化
US6795805B1 (en) Periodicity enhancement in decoding wideband signals
US8374853B2 (en) Hierarchical encoding/decoding device
CA2862715C (en) Multi-mode audio codec and celp coding adapted therefore
CN101765879B (zh) 在与itu-t g.711标准可互操作的多层嵌入式编码解码器中用于噪声整形的装备和方法
US6260009B1 (en) CELP-based to CELP-based vocoder packet translation
RU2584463C2 (ru) Кодирование звука с малой задержкой, содержащее чередующиеся предсказательное кодирование и кодирование с преобразованием
KR20090104846A (ko) 디지털 오디오 신호에 대한 향상된 코딩/디코딩
EP1158495B1 (de) Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen
US8160872B2 (en) Method and apparatus for layered code-excited linear prediction speech utilizing linear prediction excitation corresponding to optimal gains
EP0981816A1 (de) Audio-kodier-systeme und verfahren
JPH08123495A (ja) 広帯域音声復元装置
US6847929B2 (en) Algebraic codebook system and method
US20040111257A1 (en) Transcoding apparatus and method between CELP-based codecs using bandwidth extension
KR20010075491A (ko) 음성 코더 매개변수를 양자화하는 방법
JP5255575B2 (ja) レイヤード・コーデックのためのポストフィルタ
KR100499047B1 (ko) 서로 다른 대역폭을 갖는 켈프 방식 코덱들 간의 상호부호화 장치 및 그 방법
EP1431962B1 (de) Vorrichtung und Verfahren zur Breitbandcodierung von Sprachsignalen
CN101405792A (zh) 用于在音频解码器中对信号进行后处理的方法
JP3092653B2 (ja) 広帯域音声符号化装置及び音声復号装置並びに音声符号化復号装置
US6801887B1 (en) Speech coding exploiting the power ratio of different speech signal components
JP2004053676A (ja) 音声符号化装置および復号装置
KR0156983B1 (ko) 음성 부호기

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AC Divisional application: reference to earlier application

Ref document number: 1158495

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AT BE CH CY DE DK ES FI FR GB GR IE IT LI LU MC NL PT SE TR

RIC1 Information provided on ipc code assigned before grant

Ipc: 7G 10L 21/02 A

Ipc: 7G 10L 19/02 B

17P Request for examination filed

Effective date: 20050601

AKX Designation fees paid

Designated state(s): DE FR GB

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AC Divisional application: reference to earlier application

Ref document number: 1158495

Country of ref document: EP

Kind code of ref document: P

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): DE FR GB

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REF Corresponds to:

Ref document number: 60118627

Country of ref document: DE

Date of ref document: 20060518

Kind code of ref document: P

ET Fr: translation filed
PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20070108

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 16

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 17

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20170426

Year of fee payment: 17

Ref country code: FR

Payment date: 20170418

Year of fee payment: 17

Ref country code: DE

Payment date: 20170531

Year of fee payment: 17

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 60118627

Country of ref document: DE

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20180522

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180522

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181201

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20180531