EP3349480A1 - Videoanzeigevorrichtung und verfahren zu deren betrieb - Google Patents

Videoanzeigevorrichtung und verfahren zu deren betrieb Download PDF

Info

Publication number
EP3349480A1
EP3349480A1 EP17151645.3A EP17151645A EP3349480A1 EP 3349480 A1 EP3349480 A1 EP 3349480A1 EP 17151645 A EP17151645 A EP 17151645A EP 3349480 A1 EP3349480 A1 EP 3349480A1
Authority
EP
European Patent Office
Prior art keywords
mic
sound
microphones
video display
display apparatus
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP17151645.3A
Other languages
English (en)
French (fr)
Other versions
EP3349480B1 (de
Inventor
Onur ULUAG
Kagan Bakanoglu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vestel Elektronik Sanayi ve Ticaret AS
Original Assignee
Vestel Elektronik Sanayi ve Ticaret AS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vestel Elektronik Sanayi ve Ticaret AS filed Critical Vestel Elektronik Sanayi ve Ticaret AS
Priority to EP17151645.3A priority Critical patent/EP3349480B1/de
Priority to TR2017/02870A priority patent/TR201702870A2/tr
Publication of EP3349480A1 publication Critical patent/EP3349480A1/de
Application granted granted Critical
Publication of EP3349480B1 publication Critical patent/EP3349480B1/de
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R3/00—Circuits for transducers
    • H04R3/005—Circuits for transducers for combining the signals of two or more microphones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R2499/00—Aspects covered by H04R or H04S not otherwise provided for in their subgroups
    • H04R2499/10—General applications
    • H04R2499/15—Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
    • H04S7/303—Tracking of listener position or orientation

Definitions

  • the present invention relates to a video display apparatus according to claim 1 and to a method of operating a video display apparatus according to claim 12.
  • video display apparatuses such as televisions, video games machines and computer monitors
  • a separate device such as a dedicated remote control and/or smart phone.
  • a separate device such as a dedicated remote control and/or smart phone.
  • sound such as voice commands
  • a video display apparatus typically also comprises at least one loudspeaker, which itself emits sound in association with still or moving images displayed on the display screen, for example as a sound track accompanying a film or television programme or as sound effects accompanying a video game.
  • the object of the invention is solved by a video display apparatus according to claim 1.
  • the video display apparatus at least comprises a display screen, at least one loudspeaker for emitting a sound in association with at least one still or moving image displayed on the display screen, at least two spatially separated microphones, and an audio signal processing unit configured to separate the sound emitted by the loudspeaker from a sound received by the microphones.
  • a video display apparatus is relatively much larger than a portable device like a remote control unit or a smart phone, so that the at least two microphones can be positioned sufficiently far apart from each other to give good spatial resolution for discriminating sound sources from each other.
  • the audio signal processing unit can receive the sound emitted by the loudspeaker directly as an electronic signal before, during or after its emission, the sound emitted by the loudspeaker can be separated from the sound received by the microphones with a high degree of certainty and echoes can be easily identified and accounted for.
  • At least one of the microphones is preferably located adjacent the display screen, facing the same direction as the display screen. This improves the chances that at least one of the microphones will be facing a viewer of the display screen. More preferably, at least two of the microphones are located on either side of the display screen with the display screen between them, facing the same direction as the display screen. This is beneficial in increasing the horizontal resolution of the microphones.
  • At least one pair of the at least two microphones are spatially separated by at least 400 mm, more preferably by at least 500 mm, more preferably still by at least 600 mm, and most preferably by at least 700 mm from each other. This is advantageous because the spatial resolution of the microphones increases in proportion to their spatial separation.
  • the at least two microphones comprise three microphones arranged in a triangle. This is beneficial because it allows sound sources to be discriminated from each other in two dimensions. For example, if the triangle has one horizontal and one vertical side, this will give corresponding spatial resolution of sound sources in the horizontal and vertical directions.
  • the video display apparatus further comprises a sound source locating unit.
  • the sound source locating unit may locate the source of sounds, based upon the differences between the sound signals received by different ones of the at least two microphones. For example, the sound source locating unit may locate the source of sounds based on the different times of arrival of the sound from a single, common source at different ones of the at least two microphones.
  • the video display apparatus further comprises a voice recognition unit.
  • a voice recognition unit is beneficial because it can allow the video display apparatus to adopt one of a plurality of different user profiles according to the voice of a user recognized by the voice recognition unit.
  • the video display apparatus preferably further also comprises a voice command execution unit.
  • a voice command execution unit This is beneficial because it can allow the video display apparatus to be controlled by a user issuing voice commands, such as "switch to channel A", "increase volume”, and so on, without the need for a separate control device, such as a dedicated remote control or a smart phone. It also allows the video display apparatus to be used in hands-free multimedia and gaming applications.
  • the video display apparatus further comprises a multi-view display unit configured to display at least two different still or moving images on the display screen simultaneously.
  • Multi-view is an existing display technology allowing at least two different still or moving images to be displayed on the display screen simultaneously, for example by displaying the different images with different polarizations from each other.
  • a plurality of viewers with multi-view glasses of correspondingly different polarizations may then watch respective ones of the different still or moving images simultaneously without the need for a split screen. For example, one viewer may watch a film or television programme whilst another viewer browses an album of photos or plays a video game on the same display screen.
  • one or more viewers may wear head or earphones supplied by the video display apparatus with a respective sound signal appropriate to the image or images being watched by the viewer in question.
  • the voice command execution unit is preferably configured to execute a command in relation to a respective one of the simultaneously displayed still or moving images and/or a sound signal generated by the video display apparatus according to a location of a sound source issuing the command identified by the sound source locating unit.
  • the respective images may track the location of a viewer as they move.
  • one of the viewers may call out "I'm over here" as a voice command, and the video display apparatus may then redirect the displayed images and/or the accompanying sound signals accordingly.
  • the video display apparatus may further comprise a television receiver. This allows the video display apparatus to display television programmes and for the programmes to be selected and controlled using voice commands, instead of using a separate device, such as a dedicated remote control or smart phone.
  • the audio signal processing unit is further configured to separate environmental noise from the sound received by the microphones.
  • This is beneficial because it can be used to improve the accuracy of sound source location, voice recognition and execution of voice commands.
  • the separation of environmental noise from the sound received by the microphones may be carried out by sampling the sound received by the microphones at times when the loudspeaker of the video display apparatus is silent and when no rapid variations in the volume of sound received by the microphones is detected, which might otherwise be indicative of a user's voice, and then using these samples as examples of environmental noise.
  • the present invention further relates to a method of operating a video display apparatus.
  • the method at least comprises displaying on a display screen of the apparatus at least one still or moving image, emitting a sound from a loudspeaker of the apparatus in association with displaying the at least one still or moving image, receiving a sound by at least two spatially separated microphones of the apparatus, and separating the sound emitted from the loudspeaker from the sound received by the microphones.
  • the method further comprises locating at least one source of the sound received by the microphones.
  • the method further comprises recognizing at least one voice in the sound received by the microphones.
  • the method preferably further comprises executing a command issued by the at least one voice according to the location of the sound source issuing the command.
  • the present invention further relates to a computer program product or a program code or system for executing one or more than one of the herein described methods.
  • Fig. 1 schematically shows a plan view of different positions P0, P1, P2, P3 of a viewer relative to a display screen 10 of a video display apparatus. Only when the viewer is positioned somewhere in a plane equidistant between the two horizontal extremities of the display screen 10, is the viewer in a "sweetspot", as represented in Fig. 1 by position P0. In this position, a pair of spatially separated microphones, each respectively located adjacent one of the two horizontal extremities of the display screen 10, will receive the same sound emitted by the viewer as each other. In all other positions, such as those represented by P1, P2, P3 in Fig. 1 , the viewer is at a greater distance from one of the two horizontal extremities of the display screen 10 than from the other.
  • the sound received by one of the pair of spatially separated microphones located adjacent one of the horizontal extremities of the display screen 10 will be different from the sound received by the other such microphone so that it is possible to effectively separate the sound emitted by the loudspeaker from a surrounding sound received by the microphones.
  • Fig. 2 schematically represents separating and processing sound signals received from a plurality of different sound sources Source 1, Source 2, Source 3 by stereo microphones 1, 2 by means of an audio signal processing unit 20.
  • the stereo microphones 1, 2 produce left and right channel audio signals as illustrated in Fig. 2 .
  • the audio signal processing unit 20 compares these left and right channel audio signals and extracts from them estimates Estimate 1, Estimate 2, Estimate 3, each of which respectively corresponds to one of the sounds produced by Source 1, Source 2, Source 3.
  • Fig. 3 schematically represents an embodiment of a video display apparatus 100.
  • the video display apparatus 100 comprises a display screen 10 and a plurality of spatially separated microphones 1, 2, 3.
  • the microphones 1, 2, 3 are located adjacent the display screen and are arranged in a triangle.
  • the pair of microphones 1, 2 are spatially separated from each other by more than 400 mm
  • the pair of microphones 2, 3 are spatially separated from each other by more than 500 mm
  • the pair of microphones 1, 3 are spatially separated from each other by more than 600 mm.
  • the video display apparatus 100 further comprises several loudspeakers (not visible in Fig. 3 ) for emitting a sound in association with the display of at least one still or moving image on the display screen 10.
  • the video display apparatus 100 also contains a television receiver and an audio signal processing unit, neither of which are visible in Fig. 3 .
  • the audio signal processing unit is configured to separate the sound emitted by the loudspeakers from a sound received by the microphones 1, 2, 3.
  • Fig. 4 schematically represents a method of calculating the distances in three-dimensions of a plurality of spatially separated microphones, Mic 0, Mic 1, Mic 2, Mic 3 from a single source of sound, S.
  • the sound source, S is located at co-ordinates x, y, z in an arbitrarily defined three-dimensional Cartesian co-ordinate system and emits a sound at time, t.
  • the plurality of spatially separated microphones, Mic 0, Mic 1, Mic 2, Mic 3 comprises four different combinations of three microphones arranged in a triangle.
  • Mic 0 is located at co-ordinates x0, y0, z0 and receives the sound from source S at time to.
  • Mic 1 is located at co-ordinates x1, y1, z1 and receives the sound from source S at time t1.
  • Mic 2 is located at co-ordinates x2, y2, z2 and receives the sound from source S at time t2.
  • Mic 3 is located at co-ordinates x3, y3, z3 and receives the sound from source S at time t3.
  • the distance (x0 - x, y0 - y, z0 - z) of Mic 0 from the sound source S is therefore given by the speed of sound, c, multiplied by the difference between the time, t0, of reception of the sound by Mic 0 and the time, t, of its emission: c*(t0 - t).
  • the distance of Mic 1 from the sound source S is given by c*(t1-t)
  • the distance of Mic 2 from the sound source S is given by c*(t2-t)
  • the distance of Mic 3 from the sound source S is given by c*(t3 -t).
  • Fig. 5 schematically represents how two different sound signals respectively received from two different sources of sound may be modelled.
  • a first sound signal sine1 having a frequency of 10 Hz is emitted from a first source of sound and a second sound signal sine2 having a frequency of 20 Hz is emitted from a second source of sound.
  • sine1 and sine2 are both represented as having a sinusoidal waveform, although in practice, they may have any waveform and any other audio frequency or range of frequencies.
  • Sine1 and sine2 are both received by each one of a pair of spatially separated microphones.
  • the first microphone may be modelled by two amplifiers gain1, gain2 and by an adder labelled add1 in Fig. 5 .
  • Fig. 6 schematically represents two sound signals 61, 62, each respectively received by one of two spatially separated microphones from a single, common source.
  • the graph of Fig. 6 plots the amplitude, A, of the two sound signals 61, 62 on the y-axis or ordinate against time, t, on the x-axis or abscissa. As may be seen from Fig.
  • the two sound signals 61, 62 have the same frequency as each other and a similar waveform to each other (which, for the sake of this example, is a sinusoid) the amplitude A of the sound signal 61 differs from that of the sound signal 62, since the common source of the two signals 61, 62 is located further from one of the two microphones than the other.
  • Fig. 7 schematically represents sound wave power dissipation over distance.
  • the graph of Fig. 7 plots the amplitude, A, of a sound wave 71 on the y-axis or ordinate against distance, x, on the x-axis or abscissa from the emission of the sound wave 71 by a source, S, to its reception, R.
  • the amplitude, A, of the sound wave 71 progressively diminishes between the source, S, and its reception, R.
  • the power of the sound wave 71 which is proportional to the square of the amplitude, A, therefore also dissipates accordingly.
  • the present invention provides a video display apparatus at least comprising a display screen, at least one loudspeaker for emitting a sound in association with at least one still or moving image displayed on the display screen, at least two spatially separated microphones, and an audio signal processing unit configured to separate the sound emitted by the loudspeaker from a sound received by the microphones.
  • the present invention also provides a method of operating a video display apparatus, wherein the method at least comprises displaying on a display screen of the apparatus at least one still or moving image, emitting a sound from a loudspeaker of the apparatus in association with displaying the at least one still or moving image, receiving a sound by at least two spatially separated microphones of the apparatus, and separating the sound emitted from the loudspeaker from the sound received by the microphones.
  • a method allows for three-dimensional localization and separation of sound sources to receive and execute voice commands for control of a video display apparatus, such as a television, without the need for a remote control.

Landscapes

  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Controls And Circuits For Display Device (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
EP17151645.3A 2017-01-16 2017-01-16 Videoanzeigevorrichtung und verfahren zu deren betrieb Active EP3349480B1 (de)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP17151645.3A EP3349480B1 (de) 2017-01-16 2017-01-16 Videoanzeigevorrichtung und verfahren zu deren betrieb
TR2017/02870A TR201702870A2 (tr) 2017-01-16 2017-02-24 Vi̇deo görüntüleme aparati ve bunu çaliştirma metodu

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP17151645.3A EP3349480B1 (de) 2017-01-16 2017-01-16 Videoanzeigevorrichtung und verfahren zu deren betrieb

Publications (2)

Publication Number Publication Date
EP3349480A1 true EP3349480A1 (de) 2018-07-18
EP3349480B1 EP3349480B1 (de) 2020-09-02

Family

ID=57860671

Family Applications (1)

Application Number Title Priority Date Filing Date
EP17151645.3A Active EP3349480B1 (de) 2017-01-16 2017-01-16 Videoanzeigevorrichtung und verfahren zu deren betrieb

Country Status (2)

Country Link
EP (1) EP3349480B1 (de)
TR (1) TR201702870A2 (de)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11263798B2 (en) 2019-05-31 2022-03-01 Imagination Technologies Limited Multi-rendering in graphics processing units using render progression checks

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1835478A1 (de) * 2004-12-14 2007-09-19 Fujitsu Ten Limited Anzeigeeinrichtung, rahmenglied und reflexionsunterdrückungsglied
EP2923634A1 (de) * 2014-03-27 2015-09-30 Storz Endoskop Produktions GmbH Sprachsteuerungssystem für mehrere benutzer für medizinische vorrichtungen
US20160212525A1 (en) * 2015-01-15 2016-07-21 Honda Motor Co., Ltd. Sound source localization device, sound processing system, and control method of sound source localization device
US20160259305A1 (en) * 2014-08-22 2016-09-08 Boe Technology Group Co., Ltd. Display device and method for regulating viewing angle of display device

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20060016789A (ko) * 2003-05-27 2006-02-22 코닌클리케 필립스 일렉트로닉스 엔.브이. 에코 제거 시스템을 가진 확성기-마이크로폰 시스템 및에코 제거 방법
WO2014064325A1 (en) * 2012-10-26 2014-05-01 Nokia Corporation Media remixing system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1835478A1 (de) * 2004-12-14 2007-09-19 Fujitsu Ten Limited Anzeigeeinrichtung, rahmenglied und reflexionsunterdrückungsglied
EP2923634A1 (de) * 2014-03-27 2015-09-30 Storz Endoskop Produktions GmbH Sprachsteuerungssystem für mehrere benutzer für medizinische vorrichtungen
US20160259305A1 (en) * 2014-08-22 2016-09-08 Boe Technology Group Co., Ltd. Display device and method for regulating viewing angle of display device
US20160212525A1 (en) * 2015-01-15 2016-07-21 Honda Motor Co., Ltd. Sound source localization device, sound processing system, and control method of sound source localization device

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11263798B2 (en) 2019-05-31 2022-03-01 Imagination Technologies Limited Multi-rendering in graphics processing units using render progression checks

Also Published As

Publication number Publication date
TR201702870A2 (tr) 2018-07-23
EP3349480B1 (de) 2020-09-02

Similar Documents

Publication Publication Date Title
US10587979B2 (en) Localization of sound in a speaker system
US10206030B2 (en) Microphone array system and microphone array control method
US11758329B2 (en) Audio mixing based upon playing device location
US20110157327A1 (en) 3d audio delivery accompanying 3d display supported by viewer/listener position and orientation tracking
US11109151B2 (en) Recording and rendering sound spaces
CN109314834A (zh) 改进介导现实中声音对象的感知
CN102860041A (zh) 对收听者进行位置跟踪的扬声器
US20210266692A1 (en) Information processing device, information processing method, and information processing system
WO2017058192A1 (en) Suppressing ambient sounds
KR102454761B1 (ko) 영상표시장치의 동작 방법
US10979806B1 (en) Audio system having audio and ranging components
CN102057693B (zh) 内容再现装置及内容再现方法
EP3731537A1 (de) Systeme und verfahren in kachelanzeigesystemen
CN116261093B (zh) 可动态调整目标聆听点并消除环境物件干扰的音响系统
US10136216B2 (en) Action sound capture using subsurface microphones
EP3349480A1 (de) Videoanzeigevorrichtung und verfahren zu deren betrieb
CN109002272A (zh) 互动式指向性声音显示系统及互动式指向性声音显示方法
JP2009065292A (ja) 番組同時視聴システム、番組同時視聴方法及び番組同時視聴プログラム
KR102348658B1 (ko) 표시장치 및 그 구동 방법
CN206908863U (zh) Vr电视中的动态空间虚拟声处理系统
KR101505099B1 (ko) 3차원 음향 제공 시스템
KR102284914B1 (ko) 프리셋 영상이 구현되는 사운드 트랙킹 시스템
JP3255766B2 (ja) 音像定位システム
CN118590579A (zh) 控制方法及电子设备
JP2022007108A (ja) 情報処理装置、情報処理方法、及びプログラム

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190115

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20191025

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20200402

RIC1 Information provided on ipc code assigned before grant

Ipc: H04R 3/00 20060101AFI20200320BHEP

Ipc: H04S 7/00 20060101ALN20200320BHEP

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1310211

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200915

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602017022587

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201202

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201203

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20201202

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200902

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1310211

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200902

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210104

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210102

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602017022587

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20210603

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210116

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210131

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210131

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210116

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210131

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200923

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20170116

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200902

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20260123

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20260121

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: TR

Payment date: 20260114

Year of fee payment: 10