WO2017207103A1 - Système, procédé et support lisible par ordinateur permettant la gestion d'une ressource linguistique de reconnaissance d'entrée - Google Patents

Système, procédé et support lisible par ordinateur permettant la gestion d'une ressource linguistique de reconnaissance d'entrée Download PDF

Info

Publication number
WO2017207103A1
WO2017207103A1 PCT/EP2017/000653 EP2017000653W WO2017207103A1 WO 2017207103 A1 WO2017207103 A1 WO 2017207103A1 EP 2017000653 W EP2017000653 W EP 2017000653W WO 2017207103 A1 WO2017207103 A1 WO 2017207103A1
Authority
WO
WIPO (PCT)
Prior art keywords
input
recognition
typing
parameter
computing device
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/EP2017/000653
Other languages
English (en)
Inventor
Ali Reza EBADAT
Loïs RIGOUSTE
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
MyScript SAS
Original Assignee
MyScript SAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US15/215,751 external-priority patent/US11262909B2/en
Application filed by MyScript SAS filed Critical MyScript SAS
Priority to EP17731804.5A priority Critical patent/EP3465407B1/fr
Publication of WO2017207103A1 publication Critical patent/WO2017207103A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04883—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures for inputting data by handwriting, e.g. gesture or text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/02—Input arrangements using manually operated switches, e.g. using keyboards or dials
    • G06F3/023—Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
    • G06F3/0233—Character input methods
    • G06F3/0237—Character input methods using prediction or retrieval techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
    • G06F3/0488—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures
    • G06F3/04886—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser using a touch-screen or digitiser, e.g. input of commands through traced gestures by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Definitions

  • Computing devices continue to become more ubiquitous to daily life. They take the form of computer desktops, laptop computers, tablet computers, hybrid computers (2-in-ls), e-book readers, mobile phones, smartphones, wearable computers (including smartwatches, smart glasses/headsets), global positioning system (GPS) units, enterprise digital assistants (EDAs), personal digital assistants (PDAs), game consoles, and the like. Further, computing devices are being incorporated into vehicles and equipment, such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.
  • vehicles and equipment such as cars, trucks, farm equipment, manufacturing equipment, building environment control (e.g., lighting, HVAC), and home and commercial appliances.
  • Handwriting recognition in portable computing devices, such as smartphones, phablets and tablets, such as is in note taking, document annotation, mathematical equation input and calculation, music symbol input, sketching and drawing, etc.
  • Handwriting may also be input to non-portable computing devices, particularly with the increasing availability of touchscreen monitors for desktop computers and interactive whiteboards.
  • These types of input are usually performed by the user launching a handwriting input application on the computing device which accepts and interprets, either locally in the device or remotely via a communications link of the device, handwritten input on the touch sensitive surface and displays or otherwise renders this input as so-called 'digital ink'.
  • Such linguistic techniques using lexica and language models may also be used for providing word prediction and completion capabilities for typing.
  • US Patent Application Publication No. 2008/0266263 describes a collocated system in which keyboard input is recognized using linguistic techniques at a character level and handwriting input is recognized at character level only without using linguistic resources.
  • US Patent No. 7,848,917 describes a collocated system in which reduced keyboard and handwriting input are recognized using dedicated linguistic techniques.
  • a multi-modal input system of key typing, stroke typing and handwriting for example, which employs decoding or linguistic techniques for key typing, and separate dedicated linguistic resources for stroke typing and handwriting recognition.
  • linguistic resources typically require a relatively large amount of (memory) space because they need to be representative of typical uses of language, and therefore are at least as large as a dictionary.
  • N-class models Even with well-known techniques for saving space, such as grouping words together that share the same statistical context (so-called N-class models), the memory space required by a collocated system may be too excessive for many applications. Further, user experience may be adversely affected due to recognition inconsistencies when switching between one input type to another.
  • the different types of input may include handwriting input and typing input.
  • the performance characteristics may include the word recognition rate of the handwriting input, the word prediction rate of the typing input and the keystroke saving rate of the typing input.
  • the language model may be a class-based n-gram language model utilizing a lexicon built using one or more corpora related to the one or more languages.
  • the plurality of parameters may include one or more of a first parameter related to the size of the lexicon, a second parameter related to the corpora content used to build the lexicon and the n-gram model, a third parameter related to the number of n-gram sequences, a fourth parameter related to the order of the n-gram model, and a fifth parameter related to the number of classes in the class- based model.
  • the fourth and fifth parameters may be set to optimize the word prediction rate of the typing input.
  • the first parameter may be set to optimize the word recognition rate
  • the second parameter may be set to optimize the word prediction and keystroke saving rates of the typing input
  • the third parameter may be set to provide the linguistic resource with the predetermined size once the performance characteristics have been optimized.
  • a method for providing a linguistic resource for input recognition of multiple input types to computing devices includes allowing setting, in computing device memory, of a plurality of parameters of a linguistic resource which provides a language model of one or more languages, and causing recognition of input to the input interface of one or more types of a plurality of different types of input in the one or more languages using the linguistic resource.
  • the plurality of parameters are set in order to optimize performance characteristics of the recognition of each of the one or more types of recognized input while providing the linguistic resource with a pre-determined size.
  • the different types of input may include handwriting input and typing input.
  • the performance characteristics may include the word recognition rate of the handwriting input, the word prediction rate of the typing input and the keystroke saving rate of the typing input.
  • a non- transitory computer readable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for providing a linguistic resource for input recognition of multiple input types to a computing device.
  • the method includes allowing setting, in computing device memory, of a plurality of parameters of a linguistic resource which provides a language model of one or more languages, and causing recognition of input to the input interface of one or more types of a plurality of different types of input in the one or more languages using the linguistic resource.
  • the plurality of parameters are set in Order to optimize performance characteristics of the recognition of each of the one or more types of recognized input while providing the linguistic resource with a pre-determined size.
  • the different types of input may include handwriting input and typing input.
  • the performance characteristics may include the word recognition rate of the handwriting input, the word prediction rate of the typing input and the keystroke saving rate of the typing input.
  • the fourth and fifth parameters may be set to optimize the word prediction rate of the typing input.
  • the first parameter may be set to optimize the word recognition rate
  • the second parameter may be set to optimize the word prediction and keystroke saving rates of the typing input
  • the third parameter may be set to provide the linguistic resource with the predetermined size once the performance characteristics have been optimized.
  • FIG. 4 shows a schematic view of an example visual rendering of a keyboard layout in the input area for receiving typing input
  • FIG. 5 shows a schematic view of an example visual rendering of a handwriting panel in the input area for receiving handwriting input as depicted
  • FIG. 6 shows a schematic view of an example visual rendering of an input area for receiving keyboard and handwriting input
  • FIG. 7 shows a block diagram of a system for input recognition in accordance with an example of the present system and method
  • FIG. 8 shows a block diagram of a system for handwriting input recognition in accordance with an example of the present system and method
  • FIG. 9 shows a block diagram illustrating detail of the handwriting input recognition system of FIG. 8 in accordance with an example of the present system and method
  • FIG. 10 shows a block diagram of a system for typing input recognition in accordance with an example of the present system and method.
  • FIG. 1 1 shows a block diagram illustrating detail of the typing input recognition system of FIG. 10 in accordance with an example of the present system and method.
  • 'text' in the present description is understood as encompassing all alphanumeric characters, and strings thereof, in any written language and common place non-alphanumeric characters, e.g., symbols, used in written text.
  • 'non-text' in the present description is understood as encompassing freeform handwritten or hand-drawn content and rendered text and image data, as well as non- alphanumeric characters, and strings thereof, and alphanumeric characters, and strings thereof, which are used in non-text contexts.
  • the examples shown in these drawings are in a left-to-right written language context, and therefore any reference to positions can be adapted for written languages having different directional formats.
  • hand-drawing and handwriting are used interchangeably herein to define the creation of digital content by users through use of their hands either directly onto a digital or digitally connected medium or via an input tool, such as a hand-held stylus.
  • hand is used herein to provide concise description of the input techniques, however the use of other parts of a users' body for similar input is included in this definition, such as foot, mouth and eye.
  • FIG. 1 shows a block diagram of an example computing or digital device 100.
  • the computing device may be a computer desktop, laptop computer, tablet computer, hybrid computers (2-in-ls), e-book reader, mobile phone, smartphone, wearable computer, digital watch, interactive whiteboard, global positioning system (GPS) unit, enterprise digital assistant (EDA), personal digital assistant (PDA), game console, or the like.
  • the computing device 100 includes components of at least one processing element, some form of memory and input and/or output (I/O) devices. The components communicate with each other through inputs and outputs, such as connectors, lines, buses, cables, buffers, electromagnetic links, networks, modems, transducers, IR ports, antennas, or others known to those of ordinary skill in the art.
  • the illustrated example of the computing device 100 has at least one display 102 for outputting data from the computing device such as images, text, and video.
  • the display 102 may use LCD, plasma, LED, iOLED, CRT, or any other appropriate technology that is or is not touch sensitive as known to those of ordinary skill in the art.
  • At least some of the display 102 is co-located with at least one input interface 104.
  • the input interface 104 may be a surface employing technology such as resistive, surface acoustic wave, capacitive, infrared grid, infrared acrylic projection, optical imaging, dispersive signal technology, acoustic pulse recognition, or any other appropriate technology as known to those of ordinary skill in the art to receive user input.
  • the input interface 104 may be bounded by a permanent or video-generated border that clearly identifies its boundaries.
  • the computing device 100 may have a projected display capability or is able to operate with a projected display, such that the input interface is a virtual surface. Further, the display itself may be separate from and connected to the computing device.
  • the computing device 100 may include one or more additional I/O devices (or peripherals) that are communicatively coupled via a local interface.
  • the additional I O devices may include input devices such as a keyboard, mouse, scanner, microphone, touchpads, bar code readers, laser readers, radio-frequency device readers, or any other appropriate technology known to those of ordinary skill in the art.
  • the I/O devices may include output devices such as a printer, bar code printers, or any other appropriate technology known to those of ordinary skill in the art.
  • the I/O devices may include communications devices that communicate both inputs and outputs such as a modulator/demodulator (modem; for accessing another device, system, or network), a radio frequency (RF) or other transceiver, a telephonic interface, a bridge, a router, or any other appropriate technology known to those of ordinary skill in the art.
  • the local interface may have additional elements to enable communications, such as controllers, buffers (caches), drivers, repeaters, and receivers, which are omitted for simplicity but known to those of skill in the art. Further, the local interface may include address, control, 17 000653
  • the computing device 100 has operating circuitry 105.
  • FIG. 2 shows a block diagram of an example of the operating circuitry 105.
  • the operating circuitry 105 includes a processor 106, which is a hardware device for executing software, particularly software stored in a memory 108.
  • the memory 108 may be remote from the device, such as at a server or cloud-based system, which is remotely accessible by the computing device 100.
  • the memory 108 is coupled to the processor 106, so the processor 106 can read information from and write information to the memory 108.
  • the memory 108 may be integral to the processor 106.
  • the processor 106 and the memory 108 may both reside in a single ASIC or other integrated circuit.
  • the software in the memory 108 includes an operating system 1 10, an input management system 1 12 and an input recognition system 113, which may each include one or more separate computer programs. Each of these has an ordered listing of executable instructions for implementing logical functions.
  • the operating system 1 10 controls the execution of the input management system 1 12 and the input recognition system 113, or may incorporate the functions of these systems.
  • the operating system 110 may be any proprietary operating system or a commercially or freely available operating system, such as WEBOS, WINDOWS®, MAC and IPHONE OS®, LINUX, and ANDROID. It is understood that other operating systems may also be utilized.
  • the input management system 1 12 and input recognition system 1 13 of the present system and method may be provided without use of an operating system.
  • the input management system 1 12 includes one or more processing elements related to detection, management and treatment of user input.
  • the software may also include one or more applications related to input recognition, different functions, or both. Some examples of other applications include a text editor, telephone dialer, contacts directory, instant messaging facility, computer-aided design (CAD) program, email program, word processing program, web browser, and camera.
  • the input management system 1 12, and the applications include program(s) provided with the computing device 100 upon manufacture and may further include programs uploaded or downloaded into the computing device 100 after manufacture.
  • the input management system 1 12 of the present system and method manages input into the computing device 100 via the input interface 104, for example.
  • Input is managed through the provision of input tools to users and the handling of the input for processing and the like.
  • the input tools include the provision and display of dedicated input areas on the input interface 104 or the provision of the (substantially) entire input interface 104 for the receipt of user input via interaction with or in relation to the input interface 104.
  • the dimensions and functionality of these input areas are provided in correspondence with, and responsive to, the dimensions and orientation of the display area of the device display 102 in a manner well understood by those skilled in the art.
  • the illustrated layout of the keyboard panel 400 is merely an example, and many other known keyboard layouts and methods, e.g., qwerty or azerty mapped layouts for language specific variants like BoPoMoFo, Hangul, JIS, phonetic, non-qwerty layouts for different languages like Hanyu Pinyin, Jcuken, InScript, reduced keyboard, such as T9 or T12, or yet-to- be-developed keyboard layouts, are applicable to the present system and method used either singularly with respect to the computing device or selectively (discussed in detail later) by storage of different keyboard layouts in the memory 108, for example. Further, layouts that provide access to non-alphabetic characters, such as numerals, grammatical marks, emojis, etc. are also applicable, typically selectively.
  • Users may provide input with respect to the keyboard panel using a finger or some instrument such as a pen or stylus suitable for use with the input interface.
  • the user may also provide input by making a gesture above the input interface 104 if technology that senses or images motion in the vicinity of the input interface 104 is being used, or with a peripheral device of the computing device 100, such as a mouse or joystick, or with a projected interface, e.g., image processing of a passive plane surface to determine the input sequence and gesture signals.
  • the present system and method handles the user keyboard input to provide an input signal as a sequence of points that are classified as consecutive unitary key presses (e.g., key typing) or as a stroke(s) characterized by at least the stroke initiation location, the stroke termination location, and the path connecting the stroke initiation and termination locations (e.g., stroke typing) as captured by the input management system 112 and/or input recognition system 1 13. Further information such as timing, pressure, angle at a number of sample points along the path may also be captured to provide deeper detail of the keyboard strokes.
  • the display of the different input panels may also be made in accordance with the type of input made available to users by the input management system as governed by the environment of use. This can be done for example through pre-defined settings of the input management system which causes display or non-display of certain input type interfaces. For example, in an office environment, all types of input are made available to users such that the displayed user interface is that of FIG. 6.
  • Ink objects include links between the rendered display of digital ink, e.g., for handwriting, or 'typeset ink', e.g., fontified text, and the recognition candidates produced by the recognition processing, so that the displayed content is provided as interactive ink.
  • This may be achieved as described in United States Patent Application No. 15/083, 195 titled “System and Method for Digital Ink Interactivity” filed claiming a priority date of 7 January 2016 in the name of the present Applicant and Assignee, the entire contents of which is incorporated by reference herein.
  • the input management system 1 12 is configured to detect the input of typing and handwriting at the input area 300 and cause the input content (or commands) to be recognized by the input recognition system 1 13 under control of the processor 106, for example.
  • the input recognition system 1 13 and any of its components, with support and compliance capabilities, may be a source program, executable program (object code), script, application, or any other entity having a set of instructions to be performed.
  • a source program the program needs to be translated via a compiler, assembler, interpreter, or the like, which may or may not be included within the memory 108, so as to operate properly in connection with the operating system 1 10.
  • the input recognition system with support and compliance capabilities can be written as (a) an object oriented programming language, which has classes of data and methods; (b) a procedure programming language, which has routines, subroutines, and/or functions, for example but not limited to C, C++, Pascal, Basic, Fortran, Cobol, Perl, Java, Objective C, Swift, Python, C# and Ada; or (c) functional programing languages for example but no limited to Hope, Rex, Common Lisp, Scheme, Clojure, Racket, Erlang, OCaml, Haskell, Prolog, and F#.
  • an object oriented programming language which has classes of data and methods
  • a procedure programming language which has routines, subroutines, and/or functions, for example but not limited to C, C++, Pascal, Basic, Fortran, Cobol, Perl, Java, Objective C, Swift, Python, C# and Ada
  • functional programing languages for example but no limited to Hope, Rex, Common Lisp, Scheme, Clojure, Racket,
  • the input recognition system 1 13 may be a method or system for communication with an input recognition system remote from the device, such as server or cloud-based system, but is remotely accessible by the computing device 100 through communications links using the afore-mentioned communications I/O devices of the computing device 100. Further, the input management system 1 12 and the input recognition system 1 13 may operate together or be combined as a single system.
  • FIG. 7 is a schematic pictorial of an example of the input recognition system 1 13, in either its local (i.e., loaded on the device 100) or remote (i.e., remotely accessible by the device 100) forms.
  • the input recognition system 113 includes a handwriting recognition (HWR) system 1 14 as a first component and a keyboard recognition (KB ) system 1 15 as a second component.
  • HWR handwriting recognition
  • KB keyboard recognition
  • Each of these components utilize a language component 1 16 (described in detail later).
  • the input recognition system 1 13 may represent a single recognition system for both handwriting and keyboard input utilizing the language component 116.
  • the HWR system 1 14 accommodates a variety of ways in which each object may be entered whilst being recognized as the correct or intended object.
  • sequences of points entered on or via the input interface 104 are processed by the processor 106 and routed to the KBR system 1 15 for recognition processing.
  • the KBR system 1 15 accommodates a variety of ways in which each object may be entered whilst being detected as the correct or intended object.
  • FIG. 8 is a schematic pictorial of an example of the HWR system 114.
  • the HWR system 1 14 includes stages such as preprocessing 1 17, recognition 1 18 and output 120.
  • the preprocessing stage 1 17 processes the handwriting input signal ('raw' ink) to achieve greater accuracy and reduced processing time during the recognition stage 1 18.
  • This preprocessing may include normalizing of the path connecting the stroke initiation and termination locations by applying size normalization and/or methods such as B-spline approximation to smooth the input.
  • the preprocessed strokes are then passed to the recognition stage 1 18 which processes the strokes to recognize the objects formed thereby.
  • the preprocessing stage may be provided to the input recognition system 1 13 by another source, such as an optical character recognizer. Further, it is understood that the preprocessing stage may not be employed by the input recognition system 1 13 if the handwriting input signal is capable of being recognition processed without such preprocessing.
  • the recognition stage 1 18 may include different processing elements or experts.
  • FIG. 9 is a schematic pictorial of the example of FIG. 8 showing schematic detail of the recognition stage 1 18.
  • the recognition expert 124 provides classification of the features extracted by a classifier 128 and outputs a list of element candidates with probabilities or recognition scores for each node of the segmentation graph.
  • classifiers exist that could be used to address this recognition task, e.g., Support Vector Machines, Hidden Markov Models, or Neural Networks such as Multilayer Perceptrons, Deep, Convolutional or Recurrent Neural Networks. The choice depends on the complexity, accuracy, and speed desired for the task.
  • the classifier 128 incorporates information related to characteristics of handwritten characters, such as shape, slant, etc. which assists the recognition stage 1 18 in recognizing characters of candidates suggested by the other experts.
  • the language expert 126 generates linguistic meaning for the different paths in the segmentation graph using language models (e.g., grammar, semantics) of a linguistic resource.
  • the expert 126 checks the candidates suggested by the other experts according to linguistic information provided by the language component 1 16.
  • the linguistic information can include a lexicon, regular expressions, etc. and is the storage for all static data used by the language expert 126 to execute a language model.
  • Example possible language models in accordance with the present system and method are described in detail later. At this point however, it is described that a language model can rely on statistical information, such as finite state automaton (FSA), on one or. more given languages.
  • FSA finite state automaton
  • the linguistic information is substantially computed off-line, with or without adaption according to the results of recognition and user interactions, and provided to the language expert 126.
  • the language expert 126 aims at finding the best recognition path. In one example, the language expert 126 does this by exploring the language model representing the content of linguistic information. In addition to the lexicon constraint, the language expert 126 may use a language model with statistical information modeling for how frequent a given sequence of elements appears in the specified language or is used by a specific user to evaluate the linguistic likelihood of the interpretation of a given path of the segmentation graph.
  • the recognized objects are provided as the output
  • the input management system 1 12 may then render the output 120 on the display 102 as described earlier, including being included in a list of likely content candidates.
  • FIG. 10 is a schematic pictorial of an example of the KBR system 1 15.
  • the KBR system 115 includes stages such as preprocessing 130, candidate selection 132 and output 134. These stages process the detected typing points in relation to the keyboard layout being used.
  • the preprocessing stage 130 processes the typing input signal (typed ink) to achieve greater accuracy and reducing processing time during the candidate selection stage 132.
  • This preprocessing may include re-sampling/normalizing (using a background layout), smoothing, clustering of points.
  • the preprocessed sequences are then passed to the candidate selection stage 132. It is understood that the preprocessing stage may be provided to the input recognition system 1 13 by another source, such as an optical character recognizer. Further, it is understood that the preprocessing stage may not be employed by the input recognition system 113 if the keyboard input signal is capable of being recognition processed without such preprocessing.
  • the candidate selection stage 132 may include different processing elements or experts.
  • FIG. 1 1 is a schematic pictorial of the example of FIG. 10 showing schematic detail of the candidate selection stage 132. Three experts, a segmentation expert 136, a character expert 138, and a language expert 140, are illustrated which collaborate through dynamic programming to generate the output 134.
  • the segmentation expert 136 defines the different ways to segment the input signals into individual element hypotheses which form sequences of elements as a segmentation graph in accordance with layout information 142.
  • the element hypotheses are formed in sequences of mandatory points
  • the element hypotheses are formed as re-sampled sequences of optional points, e.g., the segmentation expert 136 forms hypotheses by allowing keys (e.g., characters) to be skipped on a continuous path.
  • the KBR system 1 15 may also determine the keys
  • the segmentation graph is produced with paths having nodes according to element hypotheses produced for each or some of these neighboring keys as well (thereby implementing so-called 'fuzzy' logic).
  • the character expert 138 provides probability scores for characters according to the input signal and the layout information 142 and outputs a list of element candidates with probabilities or scores for each node of the segmentation graph.
  • the layout information 142 also provides the characters or commands (such as, keyboard layout change, menu launching and editing operations on the displayed recognized content, for example) assigned to each of the keys 402/404.
  • the KBR system 1 15 determines the character(s) or functions corresponding to the keys 402/404 determined as the nodes of the segmentation graph.
  • Keyboard layout change may be provided by interaction with the input panel 400 such as input of a multiple-point gesture, like swiping, in order to 'reveal' display of different keyboard layouts.
  • a keyboard layout may provide access to alternates of the displayed character keys, such as accents, upper/lower case, language changes, symbols, numbers, etc., through multiple interactions or long-press or pressure interactions with single keys, particularly on reduced size keyboard layouts having limited keys displayed.
  • the character expert 138 adjusts the probability scores for these 'fuzzy' points and adds character alternates based on surrounding keys of the layout, keys that may or may not be skipped for stroke typing, and/or for those not directly accessible through the displayed layout (e.g., accented variants of characters, like e, e, e for e). This can be done for all detected points, e.g., for all element hypotheses representing all nodes of the segmentation graph, or for only those points that are considered 'fuzzy', e.g., the detected point is far from the center of a key.
  • the language expert 140 generates linguistic meaning for the different paths in the segmentation graph using language models (e.g., grammar, semantics) of the linguistic resource.
  • the expert 140 checks the candidates suggested by the other experts according to linguistic information provided by the language component 1 16.
  • the linguistic information can include a lexicon, regular expressions, etc., and is the storage for all static data used by the language expert 140 to execute a language model.
  • Example possible language models in accordance with the present system and method are described in detail later. At this point however, it is described that a language model can rely on statistical information, such as finite state automaton (FSA), on one or more given languages.
  • FSA finite state automaton
  • the linguistic information is substantially computed off-line, with or without adaption according to the results of recognition and user interactions, and provided to the language expert 140.
  • the language expert 140 aims at finding the best recognition path. In one example, the language expert 140 does this by exploring the language model representing the content of linguistic information. In addition to the lexicon constraint, the language expert 140 may use a language model with statistical information modeling for how frequent a given sequence of elements appears in the specified language or is used by a specific user to evaluate the linguistic likelihood of the interpretation of a given path of the segmentation graph.
  • the selected content is provided as the output 134 to the input management system 112.
  • the input management system 1 12 may then render the output 120 on the display 102 as described earlier, including being included in a list of likely content candidates.
  • the language component 1 16 for the language models used by the HW and BR systems is shared.
  • the language component 1 1 is configured as the linguistic resource for both typing and handwriting recognition. As discussed earlier, this configuration is made in such as a way as to provide a balance between the needs of the types of recognition processing and the size of the linguistic resource. This balance is provided by taking into consideration several factors related to multimodal input interpretation.
  • the recognition rate is the rate of accurate recognition of handwritten input and may be measured as the word recognition rate (WRR) with respect to HWR using language models employing a grammatical and/or semantic linguistic resource(s).
  • word completion is a mechanism of keyboard input prediction processing in which one or more remaining characters (e.g., letters) of partially typed words are predicted in order to finish the words without further typing.
  • the efficacy of correctly completed words may be measured as the word completion rate (WCR).
  • word prediction is a mechanism of keyboard input prediction processing in which at least the next word or character (e.g., grammatical mark) after a fully input or completed word is predicted, typically based on what has been typed so far.
  • the efficacy of correctly predicted words may be measured as the word prediction rate (WPR).
  • a linguistic resource for providing maximum levels of KSR and/or WPR is typically configured differently than a linguistic resource for providing maximum levels of WRR.
  • a well-configured HWR linguistic resource for use with KBR for example, it is possible that an acceptable level of WRR is undesirably sacrificed in order to obtain acceptable levels of WPR and KSR (and WCR), or vice-versa when adapting a KBR linguistic resource for HWR.
  • This may not be such a problem where the size of the linguistic resource is unimportant, since in such a case the resource can be essentially built as collocated resources with adaption, for example.
  • a substantially exhaustive list of words in a target or main language e.g., English
  • a primary language model e.g., English
  • the lexicon is built from large amounts of text, called corpora.
  • Different corpora are built from different sources of content, such as world wide web content, internet search result content, online or offline news content, encyclopedic content, messaging content, such as SMS, email, TWITTER®,
  • n an integer number
  • Classed-based n-gram language models are useful in HWR particularly due to the stroke segmentation used which basically provides prefixes for determining which classes of the N-class model are most suitable for recognition processing by considering better stroke sequences from the characteristics of the handwritten strokes. On the other hand, since typing input from a keyboard does not include characteristics like handwriting, the same corrective mechanisms may not be used. Thus, classed-based n-gram language models are not conventionally used for recognizing and interpreting keyboard input.
  • N the number of classes in the N-class model, e.g., the value of N (fifth parameter).
  • the length (e.g., order) of the n-gram sequences is not problematic for HWR since the characteristics of the handwriting may be used as described earlier to adjust for any deficiencies, whereas for word prediction in KBR having more words acting as prefixes for the next predicted word increases the accuracy of prediction.

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Character Discrimination (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un système, un procédé et un produit-programme informatique destinés à être utilisés pour une reconnaissance d'entrée de types entrées multiples sur un dispositif informatique. Le dispositif informatique est connecté à une interface d'entrée. Un utilisateur peut fournir une entrée en appliquant une pression sur l'interface d'entrée ou en faisant un geste au-dessus de l'interface d'entrée avec son doigt ou un instrument tel qu'un stylet ou un stylo. Le dispositif informatique comprend en outre un système de gestion d'entrée pour reconnaître l'entrée. Le système de gestion d'entrée est conçu pour permettre le réglage, dans la mémoire du dispositif informatique, de paramètres d'une ressource linguistique pour un modèle de langage d'une ou de plusieurs langues et entraîner la reconnaissance de l'entrée vers l'interface d'entrée des différents types d'entrée à l'aide de la ressource linguistique. Les paramètres de ressource sont réglés pour optimiser des caractéristiques de performance de reconnaissance de chaque type d'entrée tout en fournissant à la ressource linguistique la taille prédéfinie.
PCT/EP2017/000653 2016-06-02 2017-06-02 Système, procédé et support lisible par ordinateur permettant la gestion d'une ressource linguistique de reconnaissance d'entrée Ceased WO2017207103A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP17731804.5A EP3465407B1 (fr) 2016-06-02 2017-06-02 Système, procédé et support lisible par ordinateur permettant la gestion d'une ressource linguistique de reconnaissance d'entrée

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
EP16290099 2016-06-02
EP16290099.7 2016-06-02
US15/215,751 2016-07-21
US15/215,751 US11262909B2 (en) 2016-06-02 2016-07-21 System and method for input recognition linguistic resource management

Publications (1)

Publication Number Publication Date
WO2017207103A1 true WO2017207103A1 (fr) 2017-12-07

Family

ID=59093514

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2017/000653 Ceased WO2017207103A1 (fr) 2016-06-02 2017-06-02 Système, procédé et support lisible par ordinateur permettant la gestion d'une ressource linguistique de reconnaissance d'entrée

Country Status (1)

Country Link
WO (1) WO2017207103A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116348867A (zh) * 2020-10-29 2023-06-27 国际商业机器公司 记忆退化检测和改善

Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6661920B1 (en) 2000-01-19 2003-12-09 Palm Inc. Method and apparatus for multiple simultaneously active data entry mechanisms on a computer system
US6788815B2 (en) 2000-11-10 2004-09-07 Microsoft Corporation System and method for accepting disparate types of user input
US20050234722A1 (en) * 2004-02-11 2005-10-20 Alex Robinson Handwriting and voice input with automatic correction
US7250938B2 (en) 2004-01-06 2007-07-31 Lenovo (Singapore) Pte. Ltd. System and method for improved user input on personal computing devices
US20080266263A1 (en) 2005-03-23 2008-10-30 Keypoint Technologies (Uk) Limited Human-To-Mobile Interfaces
US7750891B2 (en) 2003-04-09 2010-07-06 Tegic Communications, Inc. Selective input system based on tracking of motion parameters of an input device
US7848917B2 (en) 2006-03-30 2010-12-07 Microsoft Corporation Common word graph based multimodal input
US8667414B2 (en) 2012-03-23 2014-03-04 Google Inc. Gestural input at a virtual keyboard
US20140098024A1 (en) * 2012-10-10 2014-04-10 Microsoft Corporation Split virtual keyboard on a mobile computing device
US8843845B2 (en) 2012-10-16 2014-09-23 Google Inc. Multi-gesture text input prediction
US20150286886A1 (en) 2014-04-04 2015-10-08 Vision Objects System and method for superimposed handwriting recognition technology
US9182831B2 (en) 2011-04-09 2015-11-10 Shanghai Chule (Cootek) Information Technology Co., Ltd. System and method for implementing sliding input of text based upon on-screen soft keyboard on electronic equipment
US20150356360A1 (en) 2014-04-04 2015-12-10 Myscript System and method for superimposed handwriting recognition technology

Patent Citations (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6661920B1 (en) 2000-01-19 2003-12-09 Palm Inc. Method and apparatus for multiple simultaneously active data entry mechanisms on a computer system
US6788815B2 (en) 2000-11-10 2004-09-07 Microsoft Corporation System and method for accepting disparate types of user input
US7750891B2 (en) 2003-04-09 2010-07-06 Tegic Communications, Inc. Selective input system based on tracking of motion parameters of an input device
US7250938B2 (en) 2004-01-06 2007-07-31 Lenovo (Singapore) Pte. Ltd. System and method for improved user input on personal computing devices
US20050234722A1 (en) * 2004-02-11 2005-10-20 Alex Robinson Handwriting and voice input with automatic correction
US20080266263A1 (en) 2005-03-23 2008-10-30 Keypoint Technologies (Uk) Limited Human-To-Mobile Interfaces
US7848917B2 (en) 2006-03-30 2010-12-07 Microsoft Corporation Common word graph based multimodal input
US9182831B2 (en) 2011-04-09 2015-11-10 Shanghai Chule (Cootek) Information Technology Co., Ltd. System and method for implementing sliding input of text based upon on-screen soft keyboard on electronic equipment
US8667414B2 (en) 2012-03-23 2014-03-04 Google Inc. Gestural input at a virtual keyboard
US20140098024A1 (en) * 2012-10-10 2014-04-10 Microsoft Corporation Split virtual keyboard on a mobile computing device
US8843845B2 (en) 2012-10-16 2014-09-23 Google Inc. Multi-gesture text input prediction
US20150286886A1 (en) 2014-04-04 2015-10-08 Vision Objects System and method for superimposed handwriting recognition technology
US20150356360A1 (en) 2014-04-04 2015-12-10 Myscript System and method for superimposed handwriting recognition technology

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
PERRAUD, F.; VIARD-GAUDIN, C.; MORIN, E.; LALLICAN, P.-M.: "N-Gram and N-Class Models for On Line Handwriting Recognition", SEVENTH INTERNATIONAL CONFERENCE ON DOCUMENT ANALYSIS AND RECOGNITION (ICDAR'03), 2003, pages 1053 - 1057, XP010656914 *
PERRAUD, F.; VIARD-GAUDIN, C.; MORIN, E.; LALLICAN, P.-M.: "N-Gram and N-Class Models for On Line Handwriting Recognition", SEVENTH INTERNATIONAL CONFERENCE ON DOCUMENT ANALYSIS AND RECOGNITION (ICDAR'03), 2003, pages 1053 - 1057, XP010656914, DOI: doi:10.1109/ICDAR.2003.1227818

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116348867A (zh) * 2020-10-29 2023-06-27 国际商业机器公司 记忆退化检测和改善

Similar Documents

Publication Publication Date Title
EP3465407B1 (fr) Système, procédé et support lisible par ordinateur permettant la gestion d'une ressource linguistique de reconnaissance d'entrée
EP3420442B1 (fr) Système et procédé pour une gestion de entrée multiple
US11209976B2 (en) System and method for editing input management
EP3400521B1 (fr) Système et procédé permettant une interactvité d'encre numérique
US10852938B2 (en) System and method for mixed content digital ink interactivity
EP3387582B1 (fr) Système et procédé permettant d'embellir un tracé
JP2017514248A (ja) 重畳された手書き文字認識技術システム及び重畳された手書き文字認識技術方法。
EP3491580B1 (fr) Système et procédé pour embellir une encre numérique superposée
EP3535652B1 (fr) Système et procédé de reconnaissance d'entrée par trace manuscrite
WO2017207103A1 (fr) Système, procédé et support lisible par ordinateur permettant la gestion d'une ressource linguistique de reconnaissance d'entrée
EP4047465B1 (fr) Modification d'un contenu numérique
WO2024110354A1 (fr) Définition de taille de police de caractères dans une toile non contrainte
WO2017118609A1 (fr) Système et procédé permettant une interactvité d'encre numérique

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17731804

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2017731804

Country of ref document: EP

Effective date: 20190102