EP2814264A1 - Instrument auditif pour messages vocaux hors ligne - Google Patents

Instrument auditif pour messages vocaux hors ligne Download PDF

Info

Publication number
EP2814264A1
EP2814264A1 EP13172097.1A EP13172097A EP2814264A1 EP 2814264 A1 EP2814264 A1 EP 2814264A1 EP 13172097 A EP13172097 A EP 13172097A EP 2814264 A1 EP2814264 A1 EP 2814264A1
Authority
EP
European Patent Office
Prior art keywords
message
hearing instrument
user
speech
hearing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP13172097.1A
Other languages
German (de)
English (en)
Other versions
EP2814264B1 (fr
Inventor
Brent C. Kirkwood
Brian Dam Pedersen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
GN Hearing AS
Original Assignee
GN Resound AS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by GN Resound AS filed Critical GN Resound AS
Priority to DK13172097.1T priority Critical patent/DK2814264T3/da
Priority to EP13172097.1A priority patent/EP2814264B1/fr
Priority to US13/921,178 priority patent/US9788128B2/en
Publication of EP2814264A1 publication Critical patent/EP2814264A1/fr
Application granted granted Critical
Publication of EP2814264B1 publication Critical patent/EP2814264B1/fr
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Electric hearing aids
    • H04R25/55—Electric hearing aids using an external connection, either wireless or wired
    • H04R25/554—Electric hearing aids using an external connection, either wireless or wired using a wireless connection, e.g. between microphone and amplifier or using Tcoils
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R25/00—Electric hearing aids
    • H04R25/50—Customised settings for obtaining desired overall acoustical characteristics
    • H04R25/505—Customised settings for obtaining desired overall acoustical characteristics using digital signal processing

Definitions

  • a new hearing instrument is provided with capability of presenting speech messages, such as calendar reminders, tweets, sms-messages, notifications, etc., e.g., from a user's time management and communication systems at selected points in time.
  • speech messages such as calendar reminders, tweets, sms-messages, notifications, etc., e.g., from a user's time management and communication systems at selected points in time.
  • Personal time management may be performed with a computer, e.g. using an email system with electronic calendar, to-do-lists, and notes to manage daily activities and communications. Communication may also be performed via electronic social and professional networks.
  • a user recording an event or a task to be performed also records a reminder to be displayed to the user in advance to remind the user of the upcoming event or the task to be performed.
  • notifications may be displayed on a computer indicating incoming communication, such as receipt of a new email or updates in the social or professional networks, etc.
  • Notifications and reminders typically include a sound to make the user aware of the reminder or notification. Having heard the sound, the user typically has to consult a display on a computer, tablet computer, smart phone, or mobile phone, in order to know what event or task, a particular reminder or notification relates to.
  • the user may miss one or more notifications and/or reminders.
  • a hearing instrument e.g. a hearing aid
  • a new method of communicating a message to a human wearing a hearing instrument comprising the steps of
  • a new hearing instrument system having a hearing instrument and a device, wherein the device has a central processor configured for controlling
  • the device may comprise the text-to-speech processor configured for conversion of the message into the corresponding speech message, and the central processor may be configured for controlling the transmission of the corresponding speech message to the hearing instrument.
  • the hearing instrument may comprise the text-to-speech processor.
  • the device Through the Wide-Area-Network, e.g. the Internet, the device has access to electronic time management and communication tools used by the user for communication and for storage of time management and communication information relating to the user.
  • the tools and the stored information typically reside on a remote server accessed through the Wide-Area-Network.
  • a plurality of the devices with interfaces to the Wide-Area-Network may access the tools through the Wide-Area-Network and may store the information relating to the user.
  • the device may access the Wide-Area-Network through a mobile telephone network, such as GSM, IS-95, UMTS, CDMA-2000, etc.
  • a mobile telephone network such as GSM, IS-95, UMTS, CDMA-2000, etc.
  • Each of the devices may be synchronized with the remote server when connected with the remote server through the Wide-Area-Network, e.g. according to a user defined schedule, so that the information stored in the device is consistent with the information stored in the remote server, i.e. during synchronization, the information in the remote server is updated with possible changes entered into the device by the user subsequent to the previous synchronization, e.g. the user may have entered new information, such as a new meeting in the calendar, during a period of time, when the device was not connected to the remote server; and the information in the device is also updated with possible changes entered into the remote server subsequent to the previous synchronization, e.g. another person may have send an invitation to a new meeting to the user.
  • the tools may include electronic calendar system(s), email system(s), such as Microsoft Outlook Express, Lotus notes, Windows Mail, Mozilla Thunderbird, Postbox, Apple Mail, Opera Mail, KMail, Hotmail, Yahoo, Gmail, AOL, etc, social network(s), professional network(s), such as Facebook ®, LinkedIn ®, Google+, tumblr, Twitter, MySpace, etc, RSS/Atom feeder(s), such as Bloglines, Feedly, Google Reader, My Yahoo!, NewsBlur, Netvibes, etc, news feeder(s), etc, well-known for management of daily activities and communications.
  • email system(s) such as Microsoft Outlook Express, Lotus notes, Windows Mail, Mozilla Thunderbird, Postbox, Apple Mail, Opera Mail, KMail, Hotmail, Yahoo, Gmail, AOL, etc
  • social network(s) professional network(s)
  • Facebook ® LinkedIn ®, Google+, tumblr, Twitter, MySpace, etc
  • RSS/Atom feeder(s) such as Bloglines, Feedly, Google Reader, My
  • the information may include tasks to be performed, calendar dates, such as birthdays, anniversaries, appointments, meetings, etc, contacts, websites of interest, etc.
  • the device may reside in, and may share resources with, any type of computer, tablet PC, PDA, mobile phone, smart phone, etc.
  • the device may comprise the text-to-speech processor configured to generate a corresponding speech message, such as a spoken reminder, from the information that is stored and updated using the tools.
  • the corresponding speech message may be stored as digital audio samples in an audio file in a memory in the device for subsequent transmission to the hearing instrument, e.g. upon detection of connection with the hearing instrument, possibly together with timing information, such as a date and time of day, or corresponding to a specific date and time of day, e.g. the number of seconds, minutes, hours and/or days in advance of term expiry of the recorded event or task, the reminder should be presented to the user, e.g.
  • the hearing instrument may comprise the text-to-speech processor.
  • the message may be converted to the corresponding speech message at the time of play back of the corresponding speech message to the user; or, the message may be converted to the corresponding speech message at the time of receipt of the message by the hearing instrument, and the audio samples may be stored in a memory in the hearing instrument for play back at the selected time.
  • the device may be synchronized with the remote server when connected with the remote server through the Wide-Area-Network, e.g. according to a user defined schedule.
  • the tools provide the option of specifying a reminder to be sent to the user in advance.
  • the user may select that the reminder is forwarded as an SMS and/or an email and/or displayed in a pop-up window on a computer and/or forwarded as a corresponding speech message to the hearing instrument.
  • the user may select how long time in advance, e.g. seconds, minutes, hours and/or days, the reminder is to be presented to the user, e.g. by specifying the number of seconds, minutes, hours and/or days before the term of the recorded event or task, the reminder has to be presented to the user, e.g. 3 days before a recorded birthday, or by specifying the actual date and time of day, the reminder has to be presented to the user, or by specifying the number of seconds, minutes, hours and/or days that have to elapse from data entry until presentation of the reminder to the user, etc.
  • long time in advance e.g. seconds, minutes, hours and/or days
  • the tools also provide notifications to the user of incoming communication, such as receipt of a new email, SMS, instant message, traffic announcement, etc, or receipt of updates in social or professional networks, such as Facebook, Twitter, LinkedIn, etc, RSS/Atom feeds, etc.
  • the message may include such notifications.
  • the message may also include the new incoming information, e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • new incoming information e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • examples of corresponding speech messages include reminders on, e.g. meetings, birthdays, social gatherings, journeys, to-do items, etc, and notifications on, e.g., tweets, emails, news, traffic updates, social network updates, web page updates, etc, and communication, e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • Some corresponding speech messages may be played back immediately upon receipt by the hearing instrument.
  • Corresponding speech messages to be played back immediately may be transmitted to the hearing instrument together with timing information equal to zero.
  • the corresponding speech message may be accompanied by a notification jingle, such as a personalized notification jingle.
  • the message, or the corresponding speech message may be removed automatically from the memory of the hearing instrument after play back in order to make the part of the memory occupied by the message, or corresponding speech message, available to a new message, or corresponding speech message.
  • the message, or the corresponding speech message may be kept in memory of the hearing instrument after play back in order to make it available for subsequent repeated play back.
  • the user may access the tools and the stored information from any type of computer or device that is connected to the Wide-Area-Network by logging-in to a specific account, e.g. with a username and a password.
  • the user may authenticate other devices to access the tools and the stored information without further authentication.
  • the user may have to log onto the corresponding accounts from the device.
  • the hearing instrument has an interface for reception of the message, or the corresponding speech message, from the device, and a memory for storage of the message, or the corresponding speech message.
  • the message processor is configured for, at the selected time, control play back of the corresponding speech message by transmission of the corresponding speech message to an output transducer for conversion of the corresponding speech message into an acoustic output signal for transmission towards an eardrum of the user of the hearing instrument.
  • the hearing instrument may be a hearing aid, such as a BTE, RIE, ITE, ITC, or CIC, etc, hearing aid including a binaural hearing aid; or, the hearing instrument may be a headset, headphone, earphone, ear defender, or earmuff, etc, such as an Ear-Hook, In-Ear, On-Ear, Over-the-Ear, Behind-the-Neck, Helmet, or Headguard, etc.
  • a hearing aid such as a BTE, RIE, ITE, ITC, or CIC, etc
  • hearing aid including a binaural hearing aid such as an Ear-Hook, In-Ear, On-Ear, Over-the-Ear, Behind-the-Neck, Helmet, or Headguard, etc.
  • the new hearing instrument system is a new hearing aid system with a new hearing aid having
  • the hearing instrument such as the hearing aid, may have a timer providing information on date and time of day, and the message processor may be configured for transmitting the message at a selected date and time of day.
  • the timer may be synchronized with the device, e.g. whenever data, such as the message, is transmitted to the hearing instrument.
  • the new hearing instrument system takes advantage of the fact that a user of the hearing instrument system, especially a hearing aid user, already wears the hearing instrument and therefore, the user is able to listen to corresponding speech messages without having to perform additional tasks, such as mounting a headphone or headset on his or her head, bringing a telephone to the ear, and/or looking at a screen and/or select information to be displayed and/or played back, and/or looking at a dashboard of a car and/or select information to be displayed and/or played back, etc.
  • the hearing instrument may have a wireless interface, for reception of data transmitted by the device, including messages or corresponding speech messages or distinct sounds, such as short single note tones, or distinct sequences of notes, such as notification jingles, such as a personalized notification jingles, and possibly the selected time, i.e. timing information specifying when the hearing instrument is controlled to play back the corresponding speech message.
  • messages or corresponding speech messages or distinct sounds such as short single note tones, or distinct sequences of notes, such as notification jingles, such as a personalized notification jingles, and possibly the selected time, i.e. timing information specifying when the hearing instrument is controlled to play back the corresponding speech message.
  • the hearing instrument may have a wired interface for reception of data transmitted by the device, including messages or corresponding speech messages or distinct sounds, such as short single note tones, or distinct sequences of notes, such as notification jingles, such as a personalized notification jingles, and possibly the selected time, i.e. timing information specifying when the hearing instrument is controlled to play back the corresponding speech message.
  • the wired interface may, e.g., be used during possible docking of the hearing instrument, e.g. docking for recharging of the hearing instrument.
  • the communication link may be used to synchronize the hearing instrument with the device, e.g. a timer of the hearing instrument may be synchronized with a timer of the device, and any new message; or, new messages to be presented to the user within a certain time period, e.g. within the next 24 hours, within the next week, within the next month, etc., may be transferred to the hearing instrument together with possible timing information on respective dates and times for play back of the corresponding speech messages to the user. Synchronizing data for a limited time period lowers the memory requirements of the hearing instrument.
  • the amount of available memory may be calculated and a corresponding number of new messages may be transferred to the hearing instrument together with possible timing information on respective dates and times for play back of the corresponding speech messages to the user.
  • the available memory is used to store as many messages as possible.
  • the user may use a user interface of the device to input time management and/or communication information to the tools as is well-known in the art.
  • the device may comprise the user interface, or part of the user interface, of the hearing instrument.
  • the hearing instrument may have a user interface, e.g. one or more push buttons, and/or one or more dials as is well-known from conventional hearing instruments.
  • a user interface e.g. one or more push buttons, and/or one or more dials as is well-known from conventional hearing instruments.
  • the hearing instrument system may have a user interface configured for reception of spoken user commands to control operation of the hearing instrument system.
  • the user may use the user interface of the hearing instrument to command the hearing instrument to sequentially play back the messages currently stored in the memory of the hearing instrument, e.g. in ascending or descending order of time of receipt, in ascending or descending order of time to be played back, etc, e.g. specified by the user using the user interface of the hearing instrument and/or previously specified by the user during access to the tools.
  • the user may delete messages stored in the memory, using the user interface of the hearing instrument and/or the device.
  • the user may select a new time for the message to be played back using the user interface of the hearing instrument and/or the device.
  • the new time may substitute or be added to the previous time for the message to be played back, e.g. also specified by the user using the user interface of the hearing instrument and/or the device.
  • the user may delete the time for the message to be played back without deleting the message itself from the memory of the hearing instrument using the user interface of the hearing instrument and/or the device.
  • the user may select to mute all or selected received messages using the user interface of the hearing instrument and/or the device. Subsequently, the user may select to unmute all or selected received messages using the user interface of the hearing instrument and/or the device.
  • the selected time may be a time for playing back the corresponding speech message as previously specified by the user during recording or editing of the event or task in question and transmitted to the hearing instrument together with the message for storage in the hearing instrument.
  • the corresponding speech message may be played back at more than one selected times, each of which may be transmitted to the hearing instrument together with the message in question for storage in the hearing instrument.
  • the corresponding speech message is digitized in the device into digital audio samples that is transmitted to the hearing instrument and stored in an audio file in the memory of the hearing instrument, whereby the corresponding speech message is stored in the hearing instrument in the form of an audio file.
  • the digital audio samples of the audio file is converted to an analogue audio signal in a digital-to-analogue converter of the hearing instrument and the analogue audio signal is input to an output transducer, such as a loudspeaker (termed a receiver in a hearing aid), for conversion into a corresponding acoustic speech message that is transmitted towards the eardrum of the user.
  • the transmission of messages from the device to the hearing instrument need not take place at the time at which the hearing instrument plays the corresponding speech message back. Rather, the transmission may occur anytime before the time of play back, e.g. a reminder may be transmitted to the hearing instrument together with the selected time for play back of the reminder, upon recording or editing of the reminder; whenever the hearing instrument is within receiving range of the transmitter of the device and a communication link between the device and the hearing instrument has been performed.
  • the data rate of the transmission may be slow, since the message is not streamed; rather, the data is stored in a memory in the hearing instrument for later play back.
  • data transmission may be performed whenever data transmission resources are available.
  • the communication link e.g. the wireless communication link
  • the link data rate need not be fast enough to transmit audio in real-time.
  • the corresponding speech messages may be played back to the user as high quality audio, since the corresponding speech messages may be transmitted to the user at a data rate much higher than the data rate of the communication link.
  • data transmission between the device and the hearing instrument may be performed slowly, whenever the communication link is available, and the data transmission is robust to possible communication drop outs, e.g. due to noise.
  • the transmission from the device to the hearing instrument may be performed in the background without interfering with the other desired functions of the hearing instrument.
  • Processing including signal processing, message processing, and corresponding speech message processing, in the new hearing instrument may be performed by dedicated hardware or may be performed in a signal processor, or performed in a combination of dedicated hardware and one or more signal processors.
  • processor central processor
  • messages processor signals processor
  • controller controller
  • system etc.
  • CPU-related entities either hardware, a combination of hardware and software, software, or software in execution.
  • a "processor”, “signal processor”, “controller”, “system”, etc. may be, but is not limited to being, a process running on a processor, a processor, an object, an executable file, a thread of execution, and/or a program.
  • processors central processor
  • message processor signal processor
  • controller central processor
  • a hearing instrument configured for use with a device, the hearing instrument includes: an interface for reception of a message and/or a speech message from the device, wherein the speech message is a converted form of the message and is generated using a text-to-speech processor; a memory for storage of the message and/or the speech message, and a message processor configured for, at a selected time, outputting audio samples of the speech message for transmission to a user of the hearing instrument.
  • the hearing instrument further includes the text-to-speech processor; wherein the interface is configured for reception of the message, not the speech message; and wherein the text-to-speech processor of the hearing instrument is configured to convert the message to the speech message.
  • the hearing instrument comprises a hearing aid.
  • the hearing instrument comprises a timer that is synchronized with a timer of the device, and wherein the message processor is configured for automatically outputting the audio samples at the selected time as determined with the timer.
  • the interface is also for reception of information regarding the selected time from the device.
  • the hearing instrument is a part of a hearing instrument system that includes the device.
  • the text-to-speech processor is a part of the device, and wherein the interface of the hearing instrument is configured for reception of the speech message, not the message, from the device after the text-to-speech processor of the device has converted the message to the speech message.
  • the device is configured to transmit the message and/or the speech message to the hearing instrument upon detection of a connection with the hearing instrument.
  • the device comprises: a first interface that is configured for connection with a Wide-Area-Network, a second interface configured for connection with the hearing instrument, and a central processor configured for controlling reception of information relating to the user through the Wide-Area-Network, and transmission of the message and/or the speech message to the hearing instrument based on the information.
  • the selected time is included in the information.
  • a duration of the transmission of the message to the hearing instrument is longer than a duration of the transmission of the audio samples of the speech message to the user.
  • the hearing instrument system further includes a user interface configured to receive a user command to sequentially output two or more messages stored in the memory of the hearing instrument for transmission to a user of the hearing instrument system.
  • the hearing instrument system further includes a user interface configured to receive a user command to delete a selected message in the memory of the hearing instrument.
  • the hearing instrument system further includes a user interface configured to receive a user command to repeat transmission of a selected message.
  • the hearing instrument system further includes a user interface configured to receive a user command to mute a selected message.
  • a device for use with a hearing instrument includes: a first interface that is configured for reception of information relating to a user through a Wide-Area-Network, the information comprising timing information; a second interface configured for connection with the hearing instrument; and a processor configured to control the second interface to output a message and/or a speech message to the hearing instrument based on the timing information, wherein the speech message is a converted form of the message and is generated using a text-to-speech processor.
  • a method of communicating a message includes: retrieving the message from a device with access to a Wide-Area-Network; converting the message into a corresponding speech message; storing the message and/or the corresponding speech message in a memory of a hearing instrument together with timing information, and outputting the corresponding speech message for a human at a date and time as defined by the timing information.
  • the hearing instrument system further includes the speech message, not the message, is stored in the memory of the hearing instrument.
  • Fig. 1 schematically illustrates exemplary hearing aid circuitry 10 of the new hearing instrument.
  • the illustrated new hearing aid circuitry 10 may form part of any type of hearing aid of a suitable mechanical design, e.g. to be worn in the ear canal, or partly in the ear canal, behind the ear or in the concha, such as the well-known types: BTE, ITE, ITC, CIC, etc.
  • the illustrated hearing aid circuitry 10 comprises a front microphone 12 and a rear microphone 14 for conversion of an acoustic sound signal from the surroundings into corresponding microphone audio signals 16, 18 output by the microphones 14, 16.
  • the microphone audio signals 16, 18 are digitized in respective A/D converters 20, 22 for conversion of the respective microphone audio signals 16, 18 into respective digital microphone audio signals 24, 26 that are optionally pre-filtered (pre-filters not shown) and combined in signal combiner 28, for example for formation of a digital microphone audio signal 30 with directionality as is well-known in the art of hearing aids.
  • the digital microphone audio signal 30 is input to the mixer 32 configured to output a weighted sum 34 of signals input to the mixer 32.
  • the mixer output 34 is input to a hearing loss processor 36 configured to generate a hearing loss compensated output signal 38 based on the mixer output 34.
  • the hearing loss compensated output signal 38 is input to a receiver 40 for conversion into acoustic sound for transmission towards an eardrum (not shown) of a user of the hearing aid.
  • the illustrated hearing aid circuitry 10 is further configured to receive audio signals from various devices capable of audio streaming, such as smart phones, mobile phones, radios, media players, companion microphones, broadcasting systems, such as in a public place, e.g. in a church, an auditorium, a theatre, a cinema, etc., public address systems, such as in a railway station, an airport, a shopping mall, etc., etc.
  • devices capable of audio streaming such as smart phones, mobile phones, radios, media players, companion microphones, broadcasting systems, such as in a public place, e.g. in a church, an auditorium, a theatre, a cinema, etc., public address systems, such as in a railway station, an airport, a shopping mall, etc., etc.
  • digital audio including audio samples of speech messages
  • the hearing aid e.g. from a smart phone
  • the radio receiver 44 retrieves the audio samples 46 from the received radio signal, and the time and date at which the audio samples of the speech message is to be played back to the user, possible transmitter identifiers, and possible network control signals, etc.
  • the audio samples of the speech message are stored in an audio file in the memory 48 together with the time and date, at which the audio file, i.e. the speech message, has to be played back to the user.
  • the message processor 54 controls retrieval of the audio samples from the memory 48 and forwarding of the audio samples 50 to the mixer 32.
  • the message processor 54 also sets the weights 52 with which the digital microphone audio signal 30 and the audio samples 50 are added together in the mixer 32 to form the weighted output sum 34.
  • the weights may be set so that the audio file is played back to the user while other signals input to the mixer are attenuated during play back of the audio file. Alternatively, all or some of the other signals may be muted during play back of the audio file.
  • the user may enter a command through a user interface of the hearing aid of a type well-known in the art, controlling whether the other signals are muted or attenuated.
  • the hearing aid may store more than one speech message with identical or similar time and dates to be played back; i.e. one or more speech messages may be going to be played back during ongoing play back of another speech message, whereby play back of more than one speech message may overlap fully or partly in time.
  • the hearing aid may simultaneously play back more than one speech message; i.e. one or more messages may be played back during ongoing play back of another speech message, whereby more than one speech message may be played back simultaneously or partly simultaneously.
  • each speech message is treated as a separate input to the mixer 32 added to the output of the mixer with its own weights, whereby the speech messages are transmitted to the user with substantially unchanged respective times for play back.
  • the speech messages may have assigned priorities and may be transmitted to the hearing aid together with information on the priority, e.g. an integer, e.g. larger than or equal to 1, e.g. the lower the integer, the higher the priority.
  • Alarm messages may for example have the highest priority, while traffic announcements may have the second highest priority, and possible other communications may have the lowest priority. Such messages may then be played back sequentially in the order of priority one at the time without overlaps.
  • the hearing aid may be configured to always mute one or more other signals received by the hearing aid during transmission of a speech message of highest priority towards the eardrum of the user of the hearing aid.
  • Fig. 2 schematically illustrates another exemplary hearing aid circuitry 10 of the new hearing instrument system that is identical to the hearing aid circuitry 10 of Fig. 1 and operates in a similar way except for the fact that the hearing aid circuitry 10 includes the text-to-speech processor 56 that is configured to convert messages received from the device into the speech messages that is played back to the user at the selected time previously specified during recording or editing of the information on which the message is based.
  • the message is converted into a speech message at receipt of the message, and the speech message is stored in an audio file in the memory 48.
  • the message is converted into a corresponding speech message at the selected time, i.e. at the time for play back to the user.
  • the message is stored in the memory 48 and converted into a corresponding speech message immediately upon retrieval of the message.
  • the text-to-speech processor 56 is configured to generate a speech message, such as a spoken reminder, from the text message received from the device, and the generated digital audio samples 58 are stored in an audio file in the memory 48 in the hearing aid for subsequent transmission to the mixer 32 at the selected time also received from the device and stored in the memory 48.
  • a speech message such as a spoken reminder
  • Fig. 3 schematically illustrates electronic circuitry 100 of the device of the hearing aid system, which is a smart phone.
  • the device has a user interface 120, namely a touch screen 120 as is well-known from conventional smart phones, for user control and adjustment of the device and possibly the hearing aid (not shown) interconnected with the device.
  • a user interface 120 namely a touch screen 120 as is well-known from conventional smart phones, for user control and adjustment of the device and possibly the hearing aid (not shown) interconnected with the device.
  • the user may use the user interface 120 of the smart phone 100 to input information to the tools (not shown) in a way well-known in the art.
  • the smart phone 100 may further transmit speech messages output by the text-to-speech processor 116 to the hearing aid through the audio interface 114.
  • the microphone of the hearing aid may be used for reception of spoken user commands that are transmitted to the device for reception at the interface 114 and input to the unit 118 for speech recognition and decoding of the spoken commands and outputting the decoded spoken commands as control inputs to a central processor 110.
  • the central processor 110 controls the hearing aid system to perform actions in accordance with the received spoken commands.
  • the central processor 110 also controls an Internet interface 112 configured for connection with the Internet, e.g. a Wireless Local Area Network interface, a GSM interface 122, etc, and a wired audio and data interface 114, preferably a low power wireless interface, such as the Bluetooth Low Energy wireless interface, configured for connection with the hearing aid for transmission and reception of audio samples and other data to and from the hearing aid.
  • an Internet interface 112 configured for connection with the Internet, e.g. a Wireless Local Area Network interface, a GSM interface 122, etc, and a wired audio and data interface 114, preferably a low power wireless interface, such as the Bluetooth Low Energy wireless interface, configured for connection with the hearing aid for transmission and reception of audio samples and other data to and from the hearing aid.
  • the device has access to electronic time management and communication tools used by the user for communication and for storage of time management and communication information relating to the user.
  • the tools may include electronic calendar system(s), email system(s), such as Microsoft Outlook Express, Lotus notes, Windows Mail, Mozilla Thunderbird, Postbox, Apple Mail, Opera Mail, KMail, Hotmail, Yahoo, Gmail, AOL, etc, social network(s), professional network(s), such as Facebook ®, LinkedIn ®, Google+, tumblr, Twitter, MySpace, etc, RSS/Atom feeder(s), such as Bloglines, Feedly, Google Reader, My Yahoo!, NewsBlur, Netvibes, etc, news feeder(s), etc, well-known for management of daily activities and communications.
  • email system(s) such as Microsoft Outlook Express, Lotus notes, Windows Mail, Mozilla Thunderbird, Postbox, Apple Mail, Opera Mail, KMail, Hotmail, Yahoo, Gmail, AOL, etc
  • social network(s) professional network(s)
  • Facebook ® LinkedIn ®, Google+, tumblr, Twitter, MySpace, etc
  • RSS/Atom feeder(s) such as Bloglines, Feedly, Google Reader, My
  • Reminders, notifications, and received communication may include tasks to be performed, reminders of calendar dates, such as birthdays, anniversaries, appointments, meetings, etc, notifications on receipt of new SMS or new email, new Facebook update, new tweet, new RSS feed, new traffic announcement, etc, and/or the actual item notified, e.g. the SMS itself.
  • the central processor 110 is configured to access the tools for electronic time management and communication facilitating use of the hearing instrument system to manage daily activities and communication through the Wide-Area-Network.
  • a hearing aid app (not shown) executed by the central processor 110 instructs the smart phone to forward reminders and updates and received communication from the tools to the hearing aid as speech messages in accordance with settings previously made by the user and recorded with the tools.
  • the device comprises the text-to-speech processor 116 configured for conversion of messages, such as reminders or notifications or received communication etc, into speech messages for transmission to the hearing aid.
  • the user may have a plurality of devices with internet interfaces providing access to the tools and information relating to the user, and some or all of such devices may have the text-to-speech processor 116 and the interface 114 to the hearing aid and may constitute the device disclosed above.
  • the speech message is transmitted to the hearing aid together with timing information on the date and time of day of play back of the speech message. Speech messages that are desired to play back without delay after receipt by the hearing aid may have zeroes in the transmitted date field.
  • the user accesses the tools in order to record or edit an event that requires attention or a task to be performed, the user has the option of specifying a message, namely a reminder, to be sent to the user in advance.
  • a message namely a reminder
  • the user may select that the reminder is forwarded as an SMS and/or an email and/or displayed in a pop-up window on a computer and/or is forwarded to the hearing aid as a speech message.
  • the user may select the time of presentation of the reminder to the user in several ways.
  • the user may specify the date and time of day for presentation of the reminder to the user, or the user may specify the number of seconds, minutes, hours and/or days in advance of term expiry of the recorded event or task, the reminder should be presented to the user, e.g. 3 days before a recorded birthday, or the user may specify the number of seconds, minutes, hours and/or days to elapse from data entry until presentation of the reminder to the user, etc.
  • the user also receives messages in the form of notifications on incoming communication, such as receipt of a new email, SMS, instant message, etc, or receipt of updates in social or professional networks, such as Facebook, Twitter, LinkedIn, etc, RSS/Atom feeds
  • the message may also include received information, e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • received information e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • examples of speech messages include reminders on, e.g. meetings, birthdays, social gatherings, journeys, to-do items, etc, and notifications on, e.g., tweets, emails, news, traffic updates, social network updates, web page updates, etc, and communication, e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • reminders on e.g. meetings, birthdays, social gatherings, journeys, to-do items, etc
  • notifications on e.g., tweets, emails, news, traffic updates, social network updates, web page updates, etc
  • communication e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • the speech message may be accompanied by a distinct sound, such as short single note tone, or a distinct sequence of notes, such as a notification jingle, such as a personalized notification jingle.
  • Fig. 4 schematically illustrates a new hearing aid system and its operation.
  • the Wide-Area-Network is the Internet 200
  • the hearing aid system comprises the device 100, namely a smart phone 100 operating as explained with reference to Fig. 3
  • the hearing aid 10 namely a BTE hearing aid 10 operating as explained with reference to Figs. 1 and 2 .
  • the hearing aid 10 is configured for reception of a speech message 80 from the smart phone 100.
  • the speech message 80 is a reminder of a meeting taking place at the same day at 10 o'clock.
  • the user recorded the meeting in his electronic calendar a week before, and the user also set a reminder to alert the user 15 minutes before start of the meeting, i.e. at 9.45 a.m. the same day.
  • the user recorded the meeting with a computer at work without an interface to the hearing aid 10.
  • the user has set the smart phone 100 to synchronize with the electronic calendar every half hour, whenever the smart phone is connected to the Internet through a WiFi network, and since the working place has a WiFi, the smart phone 100 was synchronized with the calendar server shortly after entry of the new meeting.
  • the user has also set the smart phone 100 to send reminders to the hearing aid 100 within 24 hours of the time at which the reminders have to be played back by the hearing aid 10.
  • the hearing aid 10 and the smart phone 100 establish a mutual communication link whenever they are within coverage of their radio transmitters. Since the user usually carries the smart phone 100 and the hearing aid 10 simultaneously, the communication link between them is usually in operation and thus, approximately at 10 am the day before the day of the meeting, the reminder is transferred as a speech message 80 to the hearing aid 10.
  • the user set the reminder to be played back to the user 15 minutes before start of the meeting. Thus, at 9.45 am, the hearing aid 10 plays back the message "remember meeting with CEO in room 1A at 10 am" to the user.
  • the reminder is deleted from the memory of the hearing aid, and if not, the reminder is played back again 5 minutes before start of the meeting and subsequently deleted from the memory of the hearing aid.
  • the spoken reminder 80 is converted from a text reminder received by the smart phone 100 from the electronic calendar system through the Internet 200.
  • the conversion to the spoken reminder takes place in a text-to-speech processor 116 in the smart phone 100.
  • the text-to-speech processor 116 provides the spoken reminder as digital audio samples that is transmitted to the hearing aid 10 and stored in an audio file in the memory of the hearing aid.
  • the digital audio samples of the audio file is converted to an analogue audio signal in a digital-to-analogue converter of the hearing aid and the analogue audio signal is input to a receiver of the hearing aid 10 that outputs the acoustic speech message to the user.
  • the user interface 120 of the smart phone 100 also constitutes a user interface of the time management and communication tools used by the user as is well-known in the art.
  • the user interface 120 of the smart phone 100 also constitutes a user interface of the hearing aid as is well-known in the art.
  • the user interface 120 of the smart phone 100 is also used for user entry of conditions specifying when a speech message in the memory of the hearing aid is to be deleted, e.g. upon play back, upon second play back, upon receipt of a specific user entry, etc.
  • the user interface 120 of the smart phone 100 is also used to set volume levels of play back of the speech messages and the volume of reproduced sounds received by the microphone(s) of the hearing aid and possible other audio sources, such as media players, TV, radio, hearing loops, etc, of the hearing aid.
  • the user may have a computer at home connected to the Internet with an interface to the hearing aid 10.
  • the home computer like the smart phone, has access to the electronic time management and communication tools used by the user for communication and for storage of time management and communication information relating to the user, and like the smart phone 100, the computer regularly may regularly synchronize with the information handled by the tools as is well-known in the art.
  • the tools may include electronic calendar system(s), email system(s), such as Microsoft Outlook Express, Lotus notes, Windows Mail, Mozilla Thunderbird, Postbox, Apple Mail, Opera Mail, KMail, Hotmail, Yahoo, Gmail, AOL, etc, social network(s), professional network(s), such as Facebook ®, LinkedIn ®, Google+, tumblr, Twitter, MySpace, etc, RSS/Atom feeder(s), such as Bloglines, Feedly, Google Reader, My Yahoo!, NewsBlur, Netvibes, etc, news feeder(s), etc, well-known for management of daily activities and communications.
  • email system(s) such as Microsoft Outlook Express, Lotus notes, Windows Mail, Mozilla Thunderbird, Postbox, Apple Mail, Opera Mail, KMail, Hotmail, Yahoo, Gmail, AOL, etc
  • social network(s) professional network(s)
  • Facebook ® LinkedIn ®, Google+, tumblr, Twitter, MySpace, etc
  • RSS/Atom feeder(s) such as Bloglines, Feedly, Google Reader, My
  • the information may include tasks to be performed, calendar dates, such as birthdays, anniversaries, appointments, meetings, etc, contacts, websites of interest, etc.
  • the hearing aid 10 and the home computer establish a mutual communication link whenever they are within coverage of their respective radio transmitters, and whenever the communication link is established, the home computer transfers speech messages to the hearing aid 10.
  • the hearing aid 10 may receive speech messages from any device with which the communication link can be established.
  • the speech messages may also be notifications on incoming communication, such as receipt of a new email, SMS, instant message, traffic update, or updates in social or professional networks, such as Facebook, Twitter, LinkedIn, etc, RSS/Atom feeds
  • the speech message may also include the received information, e.g. an email, an SMS, a post in social or professional network, a tweet, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • examples of speech messages include reminders on, e.g. meetings, birthdays, social gatherings, journeys, to-do items, etc, and notifications on, e.g., tweets, emails, news, social network updates, web page updates, etc, and communication, e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • reminders on e.g. meetings, birthdays, social gatherings, journeys, to-do items, etc
  • notifications on e.g., tweets, emails, news, social network updates, web page updates, etc
  • communication e.g. an email, an SMS, a post in social or professional network, a blog post, an RSS/Atom feed, a news feed, an instant message, etc.
  • Some speech messages may be played back immediately upon receipt by the hearing aid.
  • Speech messages to be played back immediately may be transmitted to the hearing aid together with a time and date to be played back equal to zero.
  • the speech message may be accompanied by a notification jingle, such as a personalized notification jingle.
  • the speech message, or the message may be automatically removed from the memory of the hearing aid after play back in order to make the part of the memory occupied by the, possibly spoken, message available to a new, possibly spoken, message.
  • the user may access the tools and the stored information from any computer that is connected to the Wide-Area-Network by logging-in to a specific account, e.g. with a username and a password.
  • the user may authenticate other devices to access the tools and the stored information when logged-in to the account in question.
  • the user may have to log onto the corresponding accounts from the device.
  • the hearing aid may have a timer providing information on date and time of day, and the message processor may be configured for transmitting the audio file at a selected date and time of day.
  • the timer may be synchronized with the device, e.g. whenever data is transmitted to the hearing aid.
  • the new hearing aid system takes advantage of the fact that a user of the hearing aid system, especially a hearing aid user, already wears the hearing aid and therefore, the user is able to listen to played back speech messages without having to perform additional tasks, such as mounting a headphone or headset on his or her head, bringing a telephone to the ear, looking at a screen and select information to be displayed and/or played back, looking at a dashboard of a car and select information to be displayed and/or played back, etc.
  • the hearing aid may have a wireless interface for reception of data transmitted from the device, including speech messages and possibly the selected time, i.e. timing information specifying when the hearing aid is controlled to play back the speech message.
  • the user may use a user interface of the hearing aid to command the hearing aid to sequentially play back the messages of the audio files currently stored in the memory of the hearing aid, e.g. in ascending or descending order of time of receipt, in ascending or descending order of time to be played back, etc, e.g. also specified by the user using the user interface.
  • the user may select a new time for the message to be played back using the user interface. For example, tapping a push button twice may cause the speech message to be played back again 5 minutes later.
  • the selected time may be a time for playing back the message as previously specified by the user during recording or editing of the event or task in question and transmitted to the hearing aid for storage together with the message in the hearing aid.
  • the speech message may be played back at more than one selected times, each of which may be transmitted to the hearing aid for storage with the message in question.
  • the user is relieved from the task of consulting other equipment for updates on upcoming events and incoming communication; rather, the user need not change anything or take any particular actions in order to be able to receive speech messages.
  • the transmission of messages from the smart phone 100 to the hearing aid 10 need not take place at the time at which the hearing aid plays the speech message back. Rather, the transmission may occur anytime before the time of play back, e.g. a reminder may be transmitted to the hearing aid together with the time for play back of the reminder, upon recording or editing of the reminder; whenever the hearing aid is within receiving range of the transmitter of the device.
  • the data rate of the transmission may be slow, since the message samples is not used for streaming; rather, the data is stored in a memory in the hearing aid for later play back.
  • data transmission may be performed whenever data transmission resources are available.
  • the communication link e.g. the wireless communication link
  • the link data rate need not be fast enough to transmit audio in real-time.
  • the speech messages may be played back to the user as high quality audio, since the speech messages may be read out of the memory of the hearing aid at a data rate much higher than the data rate of the communication link.
  • Data transmission to the hearing aid may be performed, slowly, when the communication link is available, and the data transmission is robust to possible communication drop outs, e.g. due to noise.
  • the synchronization may be performed in the background without interfering with the other desired functions of the hearing aid.

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Neurosurgery (AREA)
  • Otolaryngology (AREA)
  • Physics & Mathematics (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Telephonic Communication Services (AREA)
  • Telephone Function (AREA)
EP13172097.1A 2013-06-14 2013-06-14 Instrument auditif pour messages vocaux hors ligne Active EP2814264B1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
DK13172097.1T DK2814264T3 (da) 2013-06-14 2013-06-14 Høreinstrument med offline-talebeskeder
EP13172097.1A EP2814264B1 (fr) 2013-06-14 2013-06-14 Instrument auditif pour messages vocaux hors ligne
US13/921,178 US9788128B2 (en) 2013-06-14 2013-06-18 Hearing instrument with off-line speech messages

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP13172097.1A EP2814264B1 (fr) 2013-06-14 2013-06-14 Instrument auditif pour messages vocaux hors ligne

Publications (2)

Publication Number Publication Date
EP2814264A1 true EP2814264A1 (fr) 2014-12-17
EP2814264B1 EP2814264B1 (fr) 2020-05-06

Family

ID=48613506

Family Applications (1)

Application Number Title Priority Date Filing Date
EP13172097.1A Active EP2814264B1 (fr) 2013-06-14 2013-06-14 Instrument auditif pour messages vocaux hors ligne

Country Status (2)

Country Link
EP (1) EP2814264B1 (fr)
DK (1) DK2814264T3 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3836566A1 (fr) * 2019-12-13 2021-06-16 Sivantos Pte. Ltd. Procédé de fonctionnement d'un système auditif et système auditif
EP4475563A1 (fr) * 2023-12-18 2024-12-11 Oticon A/s Procédé permettant une intelligibilité de notification d'une prothèse auditive ayant un ou plusieurs événements audio et système auditif associé

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050027537A1 (en) * 2003-08-01 2005-02-03 Krause Lee S. Speech-based optimization of digital hearing devices
US20120215532A1 (en) * 2011-02-22 2012-08-23 Apple Inc. Hearing assistance system for providing consistent human speech

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7412288B2 (en) * 2004-05-10 2008-08-12 Phonak Ag Text to speech conversion in hearing systems

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050027537A1 (en) * 2003-08-01 2005-02-03 Krause Lee S. Speech-based optimization of digital hearing devices
US20120215532A1 (en) * 2011-02-22 2012-08-23 Apple Inc. Hearing assistance system for providing consistent human speech

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3836566A1 (fr) * 2019-12-13 2021-06-16 Sivantos Pte. Ltd. Procédé de fonctionnement d'un système auditif et système auditif
US11356784B2 (en) 2019-12-13 2022-06-07 Sivantos Pte. Ltd. Method of operating a hearing device system, and hearing device system
EP4475563A1 (fr) * 2023-12-18 2024-12-11 Oticon A/s Procédé permettant une intelligibilité de notification d'une prothèse auditive ayant un ou plusieurs événements audio et système auditif associé

Also Published As

Publication number Publication date
DK2814264T3 (da) 2020-07-06
EP2814264B1 (fr) 2020-05-06

Similar Documents

Publication Publication Date Title
US9788128B2 (en) Hearing instrument with off-line speech messages
US10834493B2 (en) Time heuristic audio control
US10129380B2 (en) Wearable devices for headset status and control
CN101199128B (zh) 用于提供对短消息服务的综合管理功能的方法和移动通信终端
Kim et al. A review of assistive listening device and digital wireless technology for hearing instruments
US9424843B2 (en) Methods and apparatus for signal sharing to improve speech understanding
US20090170492A1 (en) User-programmed automated communications
US12300248B2 (en) Audio signal processing for automatic transcription using ear-wearable device
US10945083B2 (en) Hearing aid configured to be operating in a communication system
US20180279048A1 (en) Binaural recording system and earpiece set
EP2814264B1 (fr) Instrument auditif pour messages vocaux hors ligne
WO2017211426A1 (fr) Procédé et système de présentation d'un ou plusieurs messages système à un utilisateur
US20220337964A1 (en) Fitting Two Hearing Devices Simultaneously
US20160072937A1 (en) Auto reminder, alarm and response system and method
EP2605541A1 (fr) Enregistreur vocal à utiliser avec un dispositif auditif
US20260059247A1 (en) Hearing device with multi-source audio reception
US12450027B2 (en) Accessory device for a hearing device
Know Widex Hearing Aids in The UK & Ireland
US12022261B2 (en) Hearing aid in-ear announcements
WO2026076636A1 (fr) Procédé de traitement de données destiné à un casque d'écoute sans fil, casque d'écoute sans fil et produit-programme informatique correspondants
US20240242704A1 (en) Systems and Methods for Optimizing Voice Notifications Provided by Way of a Hearing Device
JP2007325201A (ja) 音源分離法
CN117729287A (zh) 音频共享方法及装置、存储介质
JP2006163572A (ja) 常時接続型コミュニケーション方法、常時接続型コミュニケーションシステム、このシステムで使用される端末装置ならびにプログラム

Legal Events

Date Code Title Description
17P Request for examination filed

Effective date: 20130614

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

R17P Request for examination filed (corrected)

Effective date: 20150610

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20170829

RAP1 Party data changed (applicant data changed or rights of an application transferred)

Owner name: GN HEARING A/S

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190711

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTC Intention to grant announced (deleted)
INTG Intention to grant announced

Effective date: 20191202

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1268772

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200515

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602013068669

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: DK

Ref legal event code: T3

Effective date: 20200702

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200506

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200907

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200906

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200807

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200806

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200806

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1268772

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200506

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602013068669

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200614

26N No opposition filed

Effective date: 20210209

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200614

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200630

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200506

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240618

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DK

Payment date: 20240619

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240618

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: CH

Payment date: 20240701

Year of fee payment: 12

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20250618

Year of fee payment: 13

REG Reference to a national code

Ref country code: CH

Ref legal event code: H13

Free format text: ST27 STATUS EVENT CODE: U-0-0-H10-H13 (AS PROVIDED BY THE NATIONAL OFFICE)

Effective date: 20260127

REG Reference to a national code

Ref country code: DK

Ref legal event code: EBP

Effective date: 20250630

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20250614

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250614

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250630

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20250630