BRPI0922035A2 - SYSTEM AND METHOD FOR AUTOMATIC VOICE TO TEXT CONVERSION - Google Patents

Abstract

system and method for automatic speech-to-text conversion. Speech recognition is performed in near real-time and enhanced by event exploration and event sequencing, employing machine learning techniques that include classifiers, sets, detectors, and cascades, and using perceptual groups. Speech recognition is also enhanced using batch processing. An auto punctuation injects punctuation into recognized text strings.

Description

Descriptive Report of the Patent of Invention for "SYSTEM AND METHOD FOR AUTOMATIC VOICE TO TEXT CONVERSION". Cross Reference to Related Applications This patent application claims the benefit of US patent application serial number 12/616,723, System and Method for Automatic Speech to Text Conversion, filed November 11, 2009, and the US patent application serial number 61/113,910, Automated Speech Processors and Automated Punctuator, filed November 12, 2008, the contents of which are fully incorporated herein by this reference thereto.

Background Art Field of the Art The present invention relates generally to automatic Lt speech recognition.

More specifically, the invention relates to techniques for "%15 improving automatic speech recognition through the use of more robust and relevant aspects of the speech signal, which include temporal information and patterns derived from perceptual clusters and processing this information with the use of machine learning techniques.

Description of the Related Art Speech perception information is non-uniformly distributed in frequency, amplitude, and time.

In all respects, the voice is highly variable.

Most automated speech recognition systems extract information at evenly spaced intervals on a single scale.

In human voice perception, some voice classes are known to be distinguished by the recurrence to temporal features, but in typical state-of-the-art speech recognition systems the temporal aspects of voice are not fully explored.

Most prior art automatic speech recognition systems include a process that extracts information from the speech signal at uniform time scales (typically 10 to 15 milliseconds) using analysis frames of uniform short duration (typically 20 to 15 milliseconds). 30 milliseconds). Voice classification based on a single short-term observation vector is unreliable because the voice signal is highly dynamic and constantly in transition as multiple voice sounds are produced. In fact, longer-term standards have to be used to create usable systems.

' A method known in the art that makes longer-term patterns available is to retain a memory of a number of short-term observation vectors which are then presented simultaneously to a voice classifier. The classifiers used with this approach are often artificial neural networks or correlation patterns. While retaining a memory of short-term observation vectors brings improved results, there are several remaining problems.

First, uniform timescale sampling, common to all frame-based methods, is not synchronous with the speech signal.

x 15 So the relationship of voice events and observation frames is ; random. This results in increased variability of the extracted features and a quantization of temporal details.

Next, extraction based on uniform analysis frames is not optimal. The information used for human perception of voice sounds occurs on many different time scales. For example, the burst of a spoken "t" sound can be as short as a few milliseconds in duration whereas a vowel can be sustained for more than a second. A sequence of many short-term observations does not present the same information as long-term information, and vice versa.

Some aspects of the voice are highly variable in the temporal dimension. For example, the extent to which a vowel is sustained depends on the speaker, the rate of voice, whether the vowel is in an accented syllable or not, where in the sentence the word containing the syllable is found. This temporal variability causes voice information to move to different frames of relative observation, significantly increasing the variability of values extracted for different examples of the same voice class and making it difficult to detect meaningful patterns in memory.

Additionally, frame-based systems typically treat all frames with equal importance.

On the contrary, human perception uses the parts of the signal that have the best signal-to-noise ratio and -5 that contain the most relevant and reliable features to make the required distinctions.

Most state-of-the-art automatic speech recognition systems incorporate Hidden Markov Models.

Hidden Markov Models are stochastic state machines.

Hidden Markov Models map estimated class probabilities of observation vectors into likely sequences of hidden (unobserved) class productions. Using the Hidden Markov Models, the temporal variability problem mentioned above is addressed by allowing each non-emission state to change to itself.

Using autotransition states the + 15 time-variability is "absorbed". Unfortunately, unless the approach is modified ]; set to explicitly extract duration information, the approach removes both unwanted and desired temporal information.

The temporal relationships of voice events bring significant information to the perception of voice sounds, particularly in the discrimination of stops, fricatives and affricates.

Furthermore, robust estimation of class probabilities requires large amounts of training data.

When usage conditions differ from training conditions, probability estimates become very inaccurate leading to inferior recognition.

The characteristics used by most state-of-the-art automatic speech recognition systems are primarily derived from short-term spectral profiles.

This approach is often used because many voice sounds have some characteristic frequency spikes called formants.

A very different approach employed by other current systems is to focus on long-term trajectories of frequency bands.

In a method called TRAPs (Temporal Patterns) voice sounds are modeled as the long-term average trajectories (-1 s) of sample sounds.

Classification is performed based on the correlation of the voice signal packets with each of the TRAP models.

Some versions of this approach have reported results comparable to short-term spectral methods. These results show that information useful for identifying voice sounds is spread over time beyond -5 the boundaries of the phoneme segments. Due to the average and window used in the method, information near the center of the TRAP is emphasized over information further away. OSTRAPs capture raw trends but do not capture temporal details.

Yet another alternative approach to frame-based feature extraction is to segment the voice at the location of certain detectable signal conditions called "events". It is assumed that each segment part has a unique class identity. Usually time alignment with a model is performed by dynamic time deflection, which “% allows the trajectories of features to be projected within a common time scale. Then, on the deflected time scale the trajectory of the feature is resampled and correlated with a pattern or used as an observation for a Hidden Markov Model. The dynamic time deflection process removes much of the time variability from the voice segments. However, discovering reliable threading events presents a challenge for event-based methods. Insertions and deletions of events result in catastrophic misalignments.

Clearly there is a need in the field for improved techniques to increase the efficiency and effectiveness of automatic speech recognition.

The human perception of voice is based, in significant part, on the relative timing of events in the voice signal. The indicators for voice perception occur over several time scales and can be shifted in time from the perception itself. Changing the temporal relationships of voice events can change the perception of voice. This is demonstrated in B. Repp, et al., Perceptual Integration of Acoustic Cues for Stop, Fricative, and Affricative Manner, Journal of Experimental Psychology: Human Perception and performance 1978, Vol. 4, Num. 4, 621-637 , through perceptual experiments where the durations of silence and friction were manipulated. One such experiment introduced a small gap of silence between the words "Say" "Shop", which causes listeners to hear "Say Chop". Another example of how the relative timing of events influences perception is referred to as voice onset time, commonly abbreviated VOT. VOT is the length of time that passes from when a stop is released to when vocal cord vibration begins. VOT is an important indicator in distinguishing various stop consonants. The importance of timing also derives from the variability in the duration of voice phenomena. Some perceptible voice phenomena are very brief while others are quite long. For example, the TIMIT writing corpus of phonemically transcribed English voices has burst interrupt segments lasting less than 5 milliseconds, while "some vowel segments last longer than 500 milliseconds. Although relative timings of events are indicators It is important for perception, the most common feature extraction methods are not sensitive to the timing of speech events Almost all speech recognition and speaker applications today extract features using a segmentation approach signal based on fixed-length analysis frames scaled ahead in time by a fixed-scale dimension. Because these analysis frames are fixed in size, they are almost always significantly smaller or significantly longer than the extents of the perceptual phenomena that they try to capture.

Although easy to implement, the common approach makes feature extraction subject to the relationship between the signal and the start point of the first frame and to the arbitrary relationship between the analysis frame size and the time scale of various voice phenomena. . A frame-based speech recognition system described in S. Basu, et al., Time shift invariant speech recognition, ICSLP98, is based on twenty-five millisecond frames staggered by ten milliseconds, shifts in the initial signal and signal relationship. first frame of less than ten milliseconds caused "significant modifications F of the spectral estimates and [cepstral frequency-honey coefficients] produced by the advance which in turn results in variations of up to [ten percent] in the word error rate in the same database”.

“5 There are many sources of variation in voice signals: such as the length of the speaker's vocal tract, accentuation, voice speed, health and emotional state, as well as background noise, etc. However, the variation reported by Basu and others is entirely due to the use of a feature extraction method in which frame size and frame alignment have arbitrary relationships with the signal. The North American Patent. cane of U.S. Number 5,956,671 (deposited June 4, 1997) to Ittycheriah et al., described techniques aimed at reducing variability of characteristics brought about by the arbitrary relationship between analysis frames and speech signal. One aspect of his invention expands the variability of the training set by subjecting multiple time-shifted versions of the signal to a fixed-frame analysis process as separate training examples. They also describe a technique used in recognition time where feature values are computed by weighting the results of fixed frame analysis to multiple delayed versions of the signal.

These techniques do not completely mitigate the problems caused by extracting features using fixed frames and fixed time scales. Furthermore, expanding the number of examples increases training time and incorporates additional variability in the model that is not present in the original speech signal. Time-shifted weightings add to the complexity and may "weight out" some perceptually relevant voice features.

In U.S. Patent Number US 6,470,311 (filed October 15, 1999), for Moncur, a method of synchronous segmentation of voice pitch based on the positive zero crossings of the output of a bandpass filter with a center frequency approximately equal to the offset partially addresses the synchronization. Non-voice voice is segmented using the average distance period computed over some unspecified time frame.

It should be noted that low signal-to-noise conditions and signals with small DC signal shifts are known to cause problems for zero-crossing based segmentation.

For high-quality speech signals, the Moncur approach represents an improvement over common still-frame analysis methods during voiced speech.

Unfortunately for voiceless voice the approach reverts to arbitrary time intervals and fixed frames.

The use of fixed frames and time intervals still maintains the precise location of events such as closing and stopping unresolved bursts.

Also, no solution is provided for whispered voice.

Clearly, a solution is needed that extracts features synchronously with the events of the speech signal itself rather than through fixed uniform frames that have arbitrary and changing relationships with speech phenomena.

The segmentation technique must be applied 3 to the entire signal for both voiced and non-voiced voices.

Additionally, voice analysis must be performed over appropriate timescales for each of the particular event types that are detected.

Current typical automatic speech recognition engine waits for a detected silence to analyze and produce output because this allows for natural segmentation and therefore results in higher accuracy due to increased context.

Waiting until the end of a voice can cause the output to be delayed by anywhere from five to twenty-five seconds.

When an application has to produce near-real-time output, as is required by applications such as closed captioning in broadcast television, smaller segmentation must reduce the context available for analysis, and less accuracy is expected and produced.

For these types of applications, what is needed is high accuracy with low latency.

Summary of the invention Some embodiments of the invention relate to automatic learning of detectors and classifiers for speech recognition.

More particularly, this invention is directed to the machine learning of detectors and classifiers that focus on the most relevant and robust aspects of the speech signal for the particular detection or classification tasks in question.

a) Some embodiments of the invention involve extracting peaks or events from the voice signal that indicate notable aspects of the signal. These modalities also involve capturing the temporal relationships between events. In preferred embodiments, a weighted classifier scheme is used to extract events. Some embodiments of the invention involve building the weighted classifier scheme for use in an automatic speech recognition engine. Some embodiments of the invention involve detecting sequences of events instead of, or in addition to, detecting individual events. In some embodiments of the invention, detectors based on alternative indicators are developed. In some embodiments of the invention, IS adaptive reinforcement algorithms are used to increase recognition performance. Some embodiments of the invention include a process for reducing the complexity of sets created by adaptive reinforcement algorithms. In some embodiments of the invention, a method for automatically creating event-based detector cascades overcomes the problems of learning highly unbalanced training sets or learning to detect rare objects. The resulting cascade of detectors provides efficient detection of rare objects by eliminating most negative examples in the early stages.

In some embodiments of the invention, a process of classifying voice into perceptual groups is performed. The process then removes ambiguities between alternative perceptions.

Some embodiments of the invention involve segmenting a speech signal at perceptually important locations. This provides a means to not only extract perceptually important timings, but also to synchronize the signal analysis with speech events, thus avoiding all the problems of asynchronous fixed frame analysis. The method-

The first one performs a pre-segmentation using low-complexity filters based on certain aspects of human perception and on the voice phenomena that they are designed to detect.

These filters detect the locations of perceptible patterns that indicate start, end, bursts, glottal pulses, and other significant speech signal events.

Pre-segmentation events define intervals that are used to synchronize certain feature computations.

The feature patterns that have been extracted synchronously are further processed to create features over longer time scales and to detect even higher level perceptual events such as phoneme boundaries, syllable heads, etc.

Preferably, a high-level speech recognition system uses all of these techniques.

In some embodiments of the invention, a plurality of methods are used in a system for automatic speech recognition.

The system receives a voice input, applies one or more of the ]; processing means the voice input, decides which processing means is most correct, and provides a resulting text string.

In the presently preferred embodiments of the invention, the automatic voice recognition system is used in real-time television closed captioning and word detection environments.

Other modalities may include virtually any form of voice transcription, which include subtitling or transcription of meetings or conference calls, real-time dictation, or oral conversion of telephone messages to written form.

Some embodiments of the invention involve processing speech signals using parallel n-series of automatic speech recognition engines in burst burst mode temporarily to reduce latency.

Some embodiments of the invention involve automatic insertion of punctuation marks in unpunctuated text.

Brief Description of the Figures Figure 1 illustrates an example of a workflow for building a weighted classifier scheme for use in a processing module of an automatic speech recognition engine.

matic according to some embodiments of the invention; Figure 2 illustrates a workflow for automatically identifying regions in a plurality of speech signals that contain events in accordance with some embodiments of the invention; ns) figure 3A illustrates the time relationships of events according to some embodiments of the invention;

Figure 3B illustrates counting events that occur within time grid units in accordance with some embodiments of the invention;

Figure 3C illustrates the structure of an event-based sum map according to some embodiments of the invention;

Figure 4 illustrates a workflow 400 for creating a cascade of detectors in accordance with some embodiments of the invention;

-" Figure 5 illustrates an example of a region that contains events from all positive examples according to some modalities of the in-

: vention;

Figure 6A illustrates another example of a region in time feature space that contains events from all positive examples in accordance with some embodiments of the invention;

Figure 6B illustrates a non-aligned region containing events from all positive examples in accordance with some embodiments of the invention;

Figure 6C illustrates an example of a non-rectangular region that contains events from all positive examples according to some embodiments of the invention;

Figure 7 illustrates the relationship of the maximum geometric boundary to the tightest and loosest boundaries in a projection of a region according to some embodiments of the invention;

Figure 8A illustrates a representation of an automatic speech-to-text system according to some embodiments of the invention;

Figure 8B illustrates a representation of an automatic speech-to-text system according to some embodiments of the invention;

Figure 8C illustrates a representation of a system for event recognition and word detection according to some embodiments of the invention; Figure 9 illustrates an example of segmentations of a voice signal according to some embodiments of the invention;

Figure 10 illustrates a perceptual contrast formula used to compute perceptual change in accordance with some embodiments of the invention;

Figure 11A illustrates a circular row memory in accordance with some embodiments of the invention;

Figure 11B illustrates an updated circular row memory in accordance with some embodiments of the invention;

Figure 11C illustrates an updated circular queue memory of

- according to some embodiments of the invention; Figure 12 illustrates a segmented circular row for holding running sums in accordance with some embodiments of the invention;

Figure 13 illustrates a segmented circular row according to some embodiments of the invention;

Figure 14 illustrates a representation of a glottal pulse detector output in a small segment of voiced voice in accordance with some embodiments of the invention;

figure 15 illustrates a representation of a syllabic nucleus detector according to some embodiments of the invention;

Figure 16 illustrates a workflow to perform extraction of

—formant according to some embodiments of the invention;

Figure 17 illustrates a workflow for performing harmonic extraction in accordance with some embodiments of the invention; :

figure 18 illustrates a representation of two consecutive processing mechanisms, overlapping in time, operating in a sequence of dictions according to some embodiments of the invention; and Figure 19 illustrates a speech-to-text system that includes an automatic scorer for some embodiments of the invention.

Detailed Description of the Invention ; The invention relates to the automatic learning of detectors and classifiers for speech recognition.

More particularly, this invention is directed towards machine learning detectors and classifiers that focus on the most relevant and robust aspects of the speech signal, including temporal information for the particular detection or classification tasks in question.

In the currently preferred embodiments of the invention, the automatic speech recognition system is used in real-time television closed captioning and word detection environments.

Although automatic speech recognition has improved - over the years it still doesn't come close to human performance.

Noise levels that do not cause any difficulties for human listeners can often render automatic speech recognition systems unusable in the prior art.

Improvements in accuracy come, above all, at the cost of increased processing time and computational complexity.

In significant part, these difficulties stem from the fact that the information used by humans for voice perception is unevenly distributed in frequency, amplitude and time.

Most speech recognition systems treat all points in time as equally relevant to speech perception and make all classes of determinations based on the same set of characteristics.

Human beings, on the other hand, seem to be able to select those aspects of the voice signal that are most relevant and robust to make the distinctions necessary for perception.

Neural receptors in the ear convert the acoustic signal into temporal patterns of peaks related to their dynamic range and frequency distribution characteristics.

The peak patterns. Temporals encode information and communicate it to neurons in the brain for further processing.

The neurons and synapses that make up the brain's computational units use spike patterns to encode and communicate information to one another.

The efficiency and effectiveness of pattern recognition in human neural machinery is exceptional.

Peak encoding creates a very sparse representation of the signal.

Inspired by certain aspects of human perception, the present invention encodes information extracted from the speech signal as spikes, referred to in this document as "events". In the currently preferred embodiments of the invention, event-based extraction focuses on the salient aspects of the signal and captures the temporal relationships of these aspects.

An example of an event type would be spikes in power packets from passing frequency bands.

Peaks are the locations in the voice signal where the voice energy in each band is strongest against background noise.

The temporal distance between peaks and the temporal sequence of events are strongly related to what is being said.

Event extraction is not limited to finding the peaks of bandpass filter packets.

Other events include start and shift events generated through more complex signal analysis that includes the output of subpattern detectors.

Classifiers and detectors based on any known method can be incorporated into event patterns causing them to fire when the conditions for which they are designed are detected.

Building Relevant Automatic Detectors and Classifiers As used herein, the term "classifiers" refers to a method and apparatus that assigns class labels to vectors of features, events, and/or sequences of events.

Detectors are classifiers, which assign "present" or "absent" class labels to each feature vector, event, and/or sequence of events.

Unstressed classifiers are decision functions that perform better than chance.

Classifier sets are formed by combining the results of multiple unstressed classifiers.

Reinforcement is a method known in the art to automatically construct qualifying sets by selecting and weighting unstressed classifiers so that the decision of the set is better than the decisions of any of the unstressed classifiers. Selection is made by iteratively evaluating each unstressed classifier from a relatively large set of unstressed classifiers and choosing the one that performs best in a weighted distribution of the labeled training examples. The selected unstressed classifier is added to the set and weighted to its decision based on its error rate. The distribution weights are then adjusted to emphasize the errors made by the ensemble and the next iteration is started. Because examples that were not classified correctly are emphasized in the distribution, unstressed classifiers that tend to correct ensemble errors are added in subsequent steps and overall ensemble decisions are improved. Reinforcement has shown that it generates classifiers with good generalization characteristics. Unstressed classifiers can take any form as long as they perform better than random.

Í One method for performing time pattern classification is to sample feature trajectories at multiple fixed intervals and present all feature-time points as individual features. Typically, a fixed amount of characteristic-time points is used for classification. With a fixed number of feature-time points, the correspondence between information in one instance and that of another instance is established by defining the feature vector. i In accordance with currently preferred embodiments of the invention, a different approach is used. Because uniform sampling of feature trajectories can miss details that occur between samples and uniform sampling creates many samples that contain little discriminated information, the invention instead samples feature trajectories relative to events. Events are the points on trajectories where significant information is located. Event-based extraction creates a sparse representation of the signal. This approach requires modifying the method to define unstressed classifiers typically used in other contexts, such as image processing, because examples of a given class may have zero, one, or more than one event of a given type, therefore a method is needed to establish correspondence between information in one instance and information in another instance. Characteristic values, events, and event patterns may provide evidence that is consistent with the detector's target class, or may provide evidence to the contrary. Event types, and the temporal relationships between events, represent a significant part of the evidence for or against a target class detection. Unfortunately, exact correspondence between patterns of events in different instances of the same voice does not occur. In addition, noise can cause spurious or missing events, and voice speed can cause temporal variation in event sequences. Usually machine learning techniques are designed to use feature vectors of fixed length. With - feature vectors of fixed extent, each positive and negative training example always has a value for every feature and the correspondence between feature values for each instance is found at the same location indexed in the feature vector. Unlike values in feature vectors of fixed length, events and event patterns may or may not exist and may have somewhat different temporal relationships with each other making it difficult to determine which events in an instance correspond, to an event in another example.

The invention defines methods by which the correspondence of events and patterns of events between examples can be determined, so that temporal information can be exploited to create voiceless detectors for sets of reinforced learners.

In currently preferred embodiments of the invention a time source is associated with an event of a certain type, and the time sources of all examples are aligned. Temporal variations of events that represent a certain aspect of voice are limited by a defined interval relative to the temporal origin. For a given interval, if there is a difference in the consistency with which events (of a certain type) fall within the interval for the positive class and the negative class, the difference can be exploited to create an unstressed detector. In some modality. From this invention, examples are aligned based on the location of their syllabic head events. In some embodiments of this invention, 75 sets of two or more events are aligned with respect to one of the events within each set.

To make an unstressed detector usable based on the affirmative information associated with events, the intervals defining the unstressed detector must contain events in mostly positive instances and must contain no events in mostly negative instances. These intervals can be systematically determined by evaluating all intervals that contain events from a majority of positive examples. First, the examples are brought into general temporal correspondence through alignment based on a particular common event. Optionally, examples of different general durations can be scaled to a common extent. Consistent ranges can be efficiently discovered by first, for all examples, arranging the events from different sensors (e.g. frequency band sensors) into two dimensional spaces and recording the cumulative sum of the weighted amount of events above and to the left. of each event. Then the number of events within any rectangular range can be determined through simple differences in the accumulated weighted accounts. Unstressed detectors based on each interval that contains events for most examples are evaluated and the best detector for the current weighted distribution is retained. The composite detector is evaluated on the entire training set and the distribution weights are adjusted for the errors made.

Unstressed classifiers are added according to the above process until the detector performance is perfect in the training samples or the maximum number of iterations is reached.

Figure 1 illustrates an example of a workflow 100 to build a weighted classifier scheme for use in a

processing dual of an automatic speech recognition engine. In currently preferred embodiments of the invention, the weighted classification scheme is used in the classification module of an automatic speech recognition engine, as explained below in “5 in connection with Figure 9. Workflow 100 of Figure 1 begins storing a plurality of speech signals as a training set 101 and then extracting event patterns from the training set 102, said event patterns comprising characteristic aspects of the speech signals. Next, a sample of speech signals with corresponding event patterns is accessed 103 and aligned based on the temporal location of where the event occurred within the speech signal 104. Each signal is then optionally scaled to a duration common temporal 105. - Once the extracted signals are scaled to a common duration with common event locations, a plurality of unstressed detectors are applied to the signals and the effectiveness of each unstressed classifier is tested in your ability to detect events

106. Based on measured effectiveness, unstressed classifiers are weighted, with those that perform well receiving a high coefficient and those that perform poorly receiving a low coefficient.

107.

Next, the effectiveness of the weighting scheme is tested to determine if the weighting properly recognizes events in the training set based on a predetermined effectiveness threshold.

108. The workflow queries whether the weight properly recognizes events 109. If the weighting scheme performs properly, the workflow 100 stores the weighting scheme and terminates 110. weighting scheme does not perform properly, unstressed classifiers are added to the previously applied unstressed classifiers group 111, and the workflow iterates until the threshold level of effectiveness is reached.

The event patterns of different instances of a given voice have some similarity, however, there is no exact correspondence of events between any two voice instances. If events from different examples are given a common time of reference, such as being made relative to syllable centers, the corresponding events of different examples of a given voice will occur within a region in the time plane. - sensor. Voice is highly variable and the information most useful for perception is distributed non-uniformly in frequency, amplitude, time and time scale. Therefore, specifying regions in the time-sensor plane that contain events that contribute certain perceptual information cannot be done effectively using a single scale or constant form. However, fully evaluating all possible positions, shapes and scales of regions that may contain relevant collections or corresponding events may be computationally infeasible. Therefore, a process is defined that automatically identifies corresponding event regions useful for speech perception.

? First events of a plurality of positive training examples are made relative to a common-time reference, such! how syllable centers and events are projected onto the time-path plane. Optionally, before projection, patterns can be scaled so that their duration is equal to 1. Regions in the time-trajectory plane that contain events from a majority of positive examples are retained as potential groups of corresponding events. A list of these regions is formed and used for all subsequent steps of creating unstressed detectors.

Figure 2 illustrates an example of a workflow 200 for automatically identifying regions in a plurality of speech signals that contain event patterns in accordance with some embodiments of the invention. Workflow 200 begins by aligning a group of speech signals from a training set of speech signals relative to a common time axis 7201. Next, workflow 200 optionally scales the duration of each signal individual voice in the group to a common time duration unit 202 and which projects syllable centers of the voice signals and the event centers of the voice signals onto the common time axis

203. Finally, regions of the time axis that have a high concentration of syllable centers and event centers are identified as regions that contain 204 event patterns.

“5 In addition to the disclosed techniques for identifying regions that have a high concentration of events, the invention also involves several techniques that are employed to reject regions that are unlikely to result in robust stress detectors that include, but are not limited to, event integration mapping, application of example density constraints, rejection of redundant regions and combinations thereof. Event Integration Mapping In some embodiments of the invention, an event integration mapping process is employed to reject regions that are not likely to result in useful unstress detectors.

? A known technique in the image processing field that allows fast computation of the sum of pixel values over rectangular regions is modified to allow fast rejection of infeasible regions based on event counters in the region. In the original image processing technique the first step is to compute a "sum map" in which each cell of the map corresponds to the sum of the pixel values in the rectangular region defined by the corner in that cell and the diagonally opposite corner at the origin. After this sum map has been computed, the sum of pixels of any rectangular sub-region of the image can be determined with two operations of subtraction and one of addition. The "sum map" technique is adapted for the rapid elimination of regions that cannot contain evidence of more than a specified number of events in each grid cell of a grid superimposed on the time-trajectory plane. When a grid cell event count sum map is computed then a determination of the number of events in any rectangular region can be made using only two subtraction and one addition operations. Knowing the number of events in the region is not equivalent to

lens to know the number of examples in the region, but this sets the upper limit. Therefore any region that does not have an event counter greater than or equal to the required number of examples cannot possibly contain the required number of examples.

“5 Figures 3A to 3C illustrate the structure of an event-based sum map according to some embodiments of the invention. In figure 3A a pattern of events in the time-trajectory plane is represented. In figure 3B the event counters that occur within the limits of an overlapping grid are determined. In figure 3C a sum map is known where each cell contains the sum of the counters in the rectangular region that has the origin as a corner and the cell as the diagonally opposite corner. To determine the number of events in the four central cells of figure 3C, from the value in the upper right cell of the region in question, in this case "7", the value of the region not included on the left is subtracted, in this case "3" , as in the region not included below, in this case "4", and the subtracted region ? above at the intersection of the two subtracted regions is added back, in this case "2". This results in the number of events in the region, in this case "2" (7-3-4+2 = 2). The computational cost of determining event counters for a region of any size or shape is the same. Event Density Constraint In some other embodiments of the invention, the application of event density constraints is employed to exclude regions that are not likely to result in useful unstressed detectors. For example, minimum density constraints can optionally be applied to reject regions with an event density below a specified amount. Redundant Region Rejection In some embodiments of the invention, redundant regions that are unlikely to result in useful unstressed detectors are rejected. Regions that contain other regions but do not add additional positive events beyond those included within the contained region are not added to the region list.

Referring again to Figure 2, once the regions are identified, they form constraints that are used to generate . tone detectors.

Unstress detectors may consist of a simple test to determine whether a given instance has any events within the “5 region or not, or they may be extended to include additional restrictions based on the extent of characteristic values of positive instances that have events within the region. region.

Sequence-of-Event-Based Speech Recognition Sequences of events, in general, are more powerful discriminators in automatic speech recognition than the individual events of which they are composed.

Some embodiments of the Invention involve detecting sequences of events instead of, or in addition to, detecting individual events. - In some embodiments of the invention, a sequence of events is located as a point in hyperspace by using the intervals : (possibly scaled) in sensor-temporal space as coordinates.

To understand the concept, we consider the sequence of three events produced by a single sensor, in which the second event follows the first for two time units and the third follows the second for four time units.

The time sequence of these three events with respect to each other is represented by the coordinates (2, 4). The similarity of the temporal sequences can be judged by computing a distance function between the projected points.

For example, Euclidean distance can be used for this purpose.

In order to assess which sequences may (or may not) appear consistently in the examples, the sequences of events of a positive example are designed as already described to form a set of pattern points that represent the possible sequences that can be associated with the positive examples.

A pattern point is defined based on the coordinates of each of the points in the first example and each associated pattern point counter is set to 1. The event sequences of the rest of positive events are projected into the hyperspace points with using their ranges as coordinates in a similar way to the first example. As each point in the sequence is generated it is associated with the closest standard '-point. The sequence point is added to a list associated with that pattern point and the pattern point counter is incremented “ 5 pattern point coordinates are then adjusted to become the average values of the coordinates of its points associated examples. After all examples have been processed, pattern points with high counters represent sequences of events that are highly associated with the class. The coordinates of the standard points represent the relative centers of the regions with respect to the first event in the sequence. The size and shapes of the regions can be determined by varying the sequences of associated examples. In some embodiments of the invention, it may be desirable to merge similar sequences. Candidates for the consolidator are easily determined by its distance in the projected hyperspace.

In some embodiments of the invention the process discovers combinations of regions that detect sequences of events that frequently occur together with the target class. The usefulness of these unstressed detectors depends on their co-occurrence being less frequent when the target class is not present.

The process described in this document involves a process to discover sequences of events that provide affirmative evidence of the positive class. Evidence to the contrary also has value. To discover contrary evidence, the process described above is repeated, but this time with the negative examples. Atonic inhibitor detectors are formed based on sequences that recur in negative examples with some frequency, but never or rarely occur in positive examples.

In some embodiments of the invention, sets of unstressed detectors can be formed using an adaptive reinforcement algorithm to handle unbalanced training sets or to result in less complex detectors.

Performance Improvement Through Simplification of Boosted Sets In some embodiments of the invention, boost algorithms . Adaptive tools are used to increase recognition performance. Adaptive reinforcement algorithms involve a reiterative process of calling unstressed classifiers sequentially, testing these classifiers, and adjusting the weighting coefficients accordingly. Adaptive reinforcement algorithms create sets by adding an unstressed detector per iteration without inspecting the front and without correction of previous weights. As a result, the final assembly may be more complex than necessary.

Some embodiments of the invention include a process for reducing the complexity of sets created by adaptive reinforcement algorithms. According to these modalities after the detector reaches perfection in the training set or reaches a maximum number of iterations, then a simplification process is performed. The performance of the composite detector is iteratively compared with versions of itself. Each has a difference from its removed unstressed detectors. If removing any of the unstressed detectors improves the error rate, the removal that has the highest gain is performed, otherwise, if removing any of the unstressed detectors causes no increase in the error rate, one of these detectors is removed. The process continues until no more unstressed detectors are removed.

In other embodiments of the invention, a linear programming boost algorithm is used that updates all weights in the set as new detectors are added for building sets.

Alternative Indicator Detection Human perception of voice can rely on alternative indicators when some aspects of the voice signal are corrupted. Likewise, alternative indicators can be found in a voice sample and detected in an automatic voice recognition system.

In some embodiments of the invention, detectors based on alternative indicators are developed by following the steps mentioned above to create a detector assembly and then repeating the process to make subsequent detectors with the restriction that unstressed detectors used by previously created detectors cannot be used for . build the subsequent detectors. This will maximize detector independence. Alternative multi-indicator detectors can then be combined as a set to make a detector that is tolerant of such variation.

Automatic Set Conversion to Cascade Detectors The overall set decision is the weighted sum of the individual detectors. In the standard form of the set, all unstressed classifiers have to be evaluated to make a voice determination. In some embodiments of the invention the detector array is converted into a cascade detector which reduces the amount of unstressed detectors that have to be averaged. Ranking the unstressed detectors from strongest to weakest and analyzing the relationship between the sums at each stage and the final result can establish "early" limits that convert the set to a cascade of detectors.

The relative timing of various events contains important information for speech perception. This type of information can be explored by examining persistent patterns of corresponding events from multiple events of a given word, syllable, phoneme, etc. This analysis is challenging due to the variability in all aspects of voice and the fact that different perceptual indicators occur over different time scales.

However, as explained in this document, most machine learning classification techniques are designed to learn decisions based on fixed-dimensional vectors of homogeneous information. With event-based extraction, events do or do not occur according to signal conditions. This means that a given instance may have more or fewer events than another instance of the same syllable, word, phoneme, etc. In order to train training detectors effectively using event-based extraction, it is necessary to find out which events of an example of a syllable, word, phoneme, etc.

perceptual bearing in other examples. Later in this document, methods that automatically find the boundaries of these corresponding events are described. Methods and Techniques for Using Training Examples Automatically “ 5 to Discover Relevant Support and Contradictory Information and Determining Weights to Make a Detection Decision Event-Based Cascades for Highly Unbalanced Training Sets In some embodiments of the invention, a method for automatically creating event-based detector cascades overcomes the problems of learning from highly unbalanced training sets or learning to detect rare objects. The resulting cascades of detectors provide efficient detection of rare objects - by eliminating most negative examples in the early stages. In some embodiments of the invention, creating event-based detector cascades involves creating detectors for specific words that occur rarely. Detecting rare words is used simply to illustrate the invention and other detection applications will be readily apparent to those of ordinary skill in the art having the benefit of this disclosure. For example, some other techniques include detection of voice classes of subwords, such as specific syllables, specific phonemes, broad classes of syllables, and broad classes of phonetics. Additionally, the invention can be applied to many applications that are not related to speech recognition such as industrial process monitoring, automotive system failure detection and medical equipment monitoring.

Highly unbalanced training sets with few positive examples and many negative examples are not well handled through machine learning techniques in an attempt to minimize the amount of errors. When positive examples rarely occur, for example with a rate of 1 in 100,000,000 then a detector that fails to detect this occurrence must have a very low error rate.

(error rate = 0.00000001). However, even though it has a low error rate due to never making a false detection, it is essentially useless.

Objects that are members of a class share characteristics whose values fall within certain ranges. In this way, objects with characteristics whose values are outside these ranges can be totally rejected because they do not belong to the class. However, objects that have characteristics whose values are not completely within the range may have some characteristics whose values fall within the range associated with the class. Therefore, it may be possible to delete class membership for an object if it has a single out-of-range characteristic value. Consequently, in some embodiments of the invention, confirming class membership generally requires that all relevant feature values fall within ranges consistent with the class.

? Applied to speech recognition, event-based feature extraction creates a sparse representation that preserves the information most relevant to speech class recognition including temporal information. An example of a type of event that can be extracted is the occurrence of a peak in the packet of a certain characteristic trajectory. A characteristic path packet can, for example, be computed on the output produced when the speech signal is passed through a certain bandpass filter. When many of these trajectories are computed, the events are distributed in the time-trajectory space. All useful evidence for word-class identity is associated with events in the time-path space. When event times are made relative to a common time reference, such as syllable centers, and events from multiple instances of the same class are plotted in time-trajectory space, regions are formed that contain clusters of events. related.

The locations, shape, and scale of these regions that contain clusters are class-specific. Some of these regions will be so strongly associated with the class that all positive events in the class

whether there will be events that are within the region. As explained above, an object that does not have an event within this region can be rejected as being a member of another class. Multiple characteristic values can be associated with each event. The ranges of values for each of the “5 features associated with class positive instance events within a region form intervals in additional dimensions of space. An object must have an event with associated values within the range of every relevant feature dimension to be accepted as a member of the class. The characteristics that distinguish an object that is not of the class from all objects of the class may be different from the characteristics that distinguish another object that is not of the class from all objects of the class.

In accordance with some embodiments of the invention, these relevant considerations can be discovered automatically to create a detector. Figure 4 illustrates a workflow 400 for creating a cascade of detectors in accordance with some embodiments of the invention.

Workflow 400 starts by initializing the detector cascade to contain zero detector stages 401. Next, all regions in the time-trajectory space that contain events from all positive training examples are identified and the number of examples negative events that have events within each identified region are computed 402.

Then, for each region that contains events from all positive training examples, the definition of regions can optionally be expanded to include additional feature dimensions 403. The region boundaries for any additional dimensions are selected so that they include the full range of values from the positive examples. Then negative examples that do not include feature values within all the ranges, thus established, are rejected and the counter of negative examples included in the region is reduced accordingly 404. Additional dimensions, if any, are chosen to minimize the count of negative examples included given the number of dimensions. This means that the feature dimensions used

data for different regions are the ones that best discriminate and may vary from region to region. ' Next, the region in the list that contains events from the least amount of negative training examples is selected as a '5 stage detector cascade 405. In some embodiments of the invention, a maximum number of detector stages is predetermined.

Additionally, negative examples with no events in the selected region are eliminated from further consideration 406. Next, the workflow polls how many negative examples remain 407. If there are no more negative examples remaining, a detector cascade is created that runs perfectly in the training examples, workflow 400 provides detector 408 and para. - If there are no fewer negative examples than in the previous .:15 iteration, no further improvement can be made.

In this case, workflow 400 removes the newly added stage, transcribes imperfect detector 409, and stops.

Conversely, if there are fewer negative examples than in the previous iteration, then the workflow queries whether the maximum number of detector stages has been added 410. If the maximum number of detector stages has been added to the cascade, then the flow 400 provides an imperfect detector 411 and stops.

If there are negative examples remaining and if the maximum number of detector stages has not been reached, then workflow400 reiterates and continues to build the detector cascade by adding additional stages back to step 402. After detector cascades are created they are used according to the following method.

First, events are detected and given a common reference as done during the training process. Then, starting from the first stage of the cascade, the events in the list are evaluated to determine if any are within the region.

If any events are considered within the region, the events in the list are evaluated by subsequent stages provided that at least one event is considered within the region considered by that stage.

A Next, if the object has events within the regions of all stages of the cascade, the object is detected as a member of class “5 if Finally, if the object has no events in any of the stages, the same is rejected as a member of the class by that stage and no further processing is performed.

In these examples, axis-aligned (hyper-) rectangular regions were used.

In some other embodiments of the invention, other boundary configurations are used, such as (hyper-) spheres, or (hyper-) ellipses or mixtures of boundary shapes in different regions or in different dimensions.

In addition, rectangular (hyper-)rectangular regions that are not axis-aligned can be used.

This note applies to all references. ences to atonic detectors. 2 A1s Figures 5 through 6C illustrate various examples of training example e-Events projections onto a time-characteristic value plane in accordance with some embodiments of the invention.

Figure 5 illustrates an example of a region that contains events from all positive examples.

Figure 6A illustrates another example of a region that contains events from all positive examples.

Figure 6B illustrates a non-aligned region that contains events from all positive examples.

Figure 6C illustrates an example of a non-rectangular region that contains events from all positive examples.

Improving Generalization by Maximizing the Geometric Margin The method used to identify regions in the time-path plane results in limits that are adjusted around the positive training example events contained in the region.

When used as a detector these fitted limits should reject cases where values that are only slightly different from the training example events at the outer limits of the region.

If the limits are expanded as much as possible without covering events from additional negative examples, the detector will be able to detect cases with similar values, but beyond the range of values of any of the positive training examples in the region.

However, these slack limits to the maximum can cause false detections of cases with values that are only slightly different from the values of negative example events near the limits.

E: Generalization can be improved by adjusting each of the boundaries in the region to maximize the geometric margin between detected positive instance events and rejected negative instance events.

The maximum geometric margin limits are midway between the adjusted minimum limits and the loose maximum limits.

Maximizing the geometric margins provides the best opportunity for generalization to hidden cases in training examples.

Figure 7 illustrates the relationship of the maximum geometric limit to the maximally fitted and maximally loose limits in a projection of a region. is The Use of Reliable General Category Sequences to Constrain Perceptions: Typical speech recognition systems work by recognizing details, such as phoneme or subphoneme classes, and using these details to determine speech patterns. highest level, such as words.

These low-level details are not distinguished with certainty, instead probability estimates are made for each of the classes given an observation vector of characteristic values.

Hidden Markov Models (HMM) use class probability estimates together with transition probabilities to compute the most likely sequence of intended speech sounds.

While the "build from the detail" approach is popular and reasonably effective, it does not result in automated speech recognition systems that rival human performance.

One of the drawbacks of this approach is the fact that detailed classifications are not very reliable and need to be corrected by applying higher context levels.

Furthermore, detailed classifications are highly context-dependent, but the context is not known when determining the identity of voice classes.

Additionally, the context can be represented inaccurately or with low reliability. Furthermore, accurate statistics are difficult to estimate for details in contexts that rarely occur. Variations in acoustic conditions or speech that are not represented in the model's statistical distributions cause statistical estimates to be inaccurate. Finally, the large search space for alternative solutions can be computationally intractable. Typically the search is reduced by arbitrary means such as retaining only the most likely "n". The objects of the present invention are to overcome the problems and limitations inherent in the common approach.

In general, sorting into broad categories can be performed more safely than sorting into detailed categories. For example, distinguishing between a fish and a bird can be done more reliably than determining the specific types of birds or fish. Likewise, in the case of speech recognition, broad categorization p 15 —can be performed more safely than detailed categorization.

Í Additionally, human perception seems to operate mainly on broad categorizations and consider details only when there is a reason to focus on them. In a continuous fluent voice, words are only rarely produced as the dictionary says they should be, but this causes little problem for human listeners as long as sufficient evidence is present to support a perception. Consequently, human hearers can tolerate substitutions and omissions as long as aspects of the voice fall within the broad expected reliable categories generally following the expected timing of the voice.

For example, consider the question and answer: "Why you cr-yin?", "See hit honey." The question omitted the word "are" and replaced the syllable "in" with "ing". None of these changes have much effect on human perception. Likewise, the answer is more likely to be perceived as '"Shehitmel!" even though the required "sh" sound has become similar to the "s" sound. The substitution and omission of details in these examples has little effect on perception and should probably go unnoticed by a human being. It seems that the sequence patterns : of broad categories of syllables are sufficient to index per- units. perceptions that in many cases lead to an ambiguous perception without requiring specific identification of detailed classes. “5 The invention is based on the following observations: * To a large extent, the sequence pattern of broad categories of speech can limit the possible perceptual alternatives. The perceptual alternatives form a perceptual grouping.

e The broad voice categories string itself can be used to directly access the list of perceptual alternatives.

* Additional computational effort is applied only when necessary to differentiate between the remaining alternatives within a perceptual cluster.

- * Because the alternatives in the cluster are known. 15 in training time, for each perceptual cluster the differentiation process can be optimized for maximum security or minimum computational effort. Consequently, the safest distinctions under any circumstances can be applied. This means that information from various sources can be applied, including word statistics, prosodics, grammar, etc.

and “When differentiating between alternate perceptions, the phonetics and word contexts of the alternatives are known, thus limiting the computations to distinguish features to those that are relevant and safer. In addition, context-specific detectors and classifiers can be used for added security.

According to these modalities, it is only when the sequence patterns of broad categories do not completely differentiate perception, that recourse to detail is required. Even in this case it is possible to preferentially use detailed breakdowns that are known to be more secure than other detailed breakdowns. For example, consider a sequence pattern of broad syllable categories indexed to two perceptions that were distinguishable from each other by different phonemics.

33/57 . but in two locations.

If one of the phoneme pairs was recognized as being more reliably distinguished than the other, the distinction must

' be made in the safest classification.

Likewise, context is very important for perception.

If the Good response in the example given earlier had been "cuz see hit me!", it could be perceived as "cause, he hit me!". The details of the "see" segment have not changed, but perception does not depend on the details of that segment.

In some embodiments of the invention, a unique algorithm is used to classify voice into perceptual clusters and differentiate between alternative perceptions through optimally accessing available information.

According to these modalities, at each time step (i.e. arrival of another syllable pattern or null syllable if no voice occurs within a certain time), the algorithm classifies the voice within a — 15 sequence of broad patterns but reliable, such as broad syllable categories.

Then, each broad category is associated with a category number.

Preferably, similar categories are assigned similar numbers.

The algorithm then maps broad category sequences into perceptual patterns using the category numbers as state space coordinates.

Each point in state space is associated with a perceptual grouping and differentiation strategy.

The differentiation strategy, established during training, is a sequence of steps to be performed when the perceptual cluster is accessed.

The purpose of the differentiation strategy is to differentiate between alternative perceptions to optimally access available information.

The differentiation strategy is determined by evaluating computational requirements and the success of various differentiating techniques applied in different orders and in different combinations.

The end result of applying the strategy is a reduction of alternative perceptions to a small amount, preferably one.

If the alternatives are reduced to a single perception, the

ception is activated. In a speech-to-text system this involves providing the words that correspond to the perception. In a system controlled by VOICE, actions associated with perception must be performed. If the alternatives are not reduced to a single perception and “5 a maximum latency threshold has been reached, the most likely perception is accepted as the perception and actions are generated accordingly. If the maximum latency threshold has not been reached, the remaining alternative insights are retained and interact with subsequent steps in time both to help differentiate perceptions at these steps and to be differentiated through the information available at these steps in time. Automatic Speech Recognition Mechanism In the presently preferred embodiments of the invention, an apparatus is provided for carrying out all original aspects of the invention. In the presently preferred embodiments of the invention, the automatic speech recognition system is used in closed captioning of real-time television and word detection environments.

Figure 8A illustrates a representation of an automated speech-to-text system 800 that comprises event-based extraction and recognition on a syllable scale of broad syllable classifications.

The 800 automatic text-to-speech system uses syllable-wide classification sequence patterns to index into lists of perceptual units with reference to phoneme level of detail only when necessary for differentiation. In currently preferred embodiments of the invention, the automatic text-to-speech system 800 chooses which phoneme classifications to make or chooses other differentiation methods to employ based on the security of those classifications or methods.

The automatic text-to-speech system 800 includes an acoustic analyzer 802. The acoustic analyzer receives an input speech signal 801 and digitizes said input signal 801. The acoustic analyzer 802 is coupled — optionally with a prosodic analyzer 803 and with a event extractor 804. In some embodiments of the invention, the digitized signal is processed by the prosodic analyzer 803, thereby extracting various characters.

the speaker's linguistic characteristics that include, but are not limited to, rhythm, tension, intonation, or other prosodic information that reflects: the speaker's emotional state; whether the expression is a statement, question or command; irony; sarcasm; emphasis; focus; etc. According to these modalities, the “5 prosodic information and the digitized signal are sent to the event extractor 804.

Event extractor 804 comprises a processing mechanism for automatically identifying regions in a plurality of speech signals that contain event patterns and extracting said events for speech recognition. In the currently preferred embodiments of the invention, the processes and methods described above for event recognition and extraction are employed by the event extractor 804. The event extractor 804 is coupled to a short-term event memory 805 to store the extracted voice events. The short-term event memory 805 is coupled with a plurality of event-to-text processing modules to use the extracted events to provide a resulting text stream. In currently preferred embodiments of the invention, the event stream processing modules for text comprise a syllable head detector 806, a syllable categorizer 807, a syllable sequence perceptual indexing module 808, and a categorization module. subsyllable detail 809. Event stream processing modules for text provide a text stream with added prosodic information 811 embedded in it.

The automatic text-to-speech system 800 shown in Figure 8A comprises an example of an apparatus for automatic speech recognition and for improving the same. It will be readily apparent to those skilled in the art having the benefit of this disclosure that any number of systems, configurations, hardware components, etc. can be used to perform these methods and processes for automatic speech recognition and to improve the same.

Figure 8B illustrates a representation of a speech-to-text system 820 comprising a speech recognition engine 824 for processing an input speech signal 821 in accordance with some embodiments of the invention. In the currently preferred modalities. Further to the invention, an acoustic analyzer 822 receives the input speech signal 821 and digitizes said input speech signal 821. The acoustic analyzer 822 is coupled with the prosodic analyzer 823 and with a speech recognition mechanism. 824. In some embodiments of the invention, the digitized signal is processed through prosodic analyzer 823, thereby extracting prosodic information, as explained above. In presently preferred embodiments of the invention, speech recognition engine 824 comprises a plurality of processing modules for performing various steps of speech recognition processing. As shown, the speech recognition mechanism 824 comprises: an event extractor 825; an identifier of . default 826; an unstressed region filter 827; a forced set simplifier 828; an 829 event sequence identifier; an alternative indicator detector 830; a cascade detector array creator 831; an 832 speech generalizer; and a perceptual cluster differentiation module 833. Although specific processing modules are listed in this document, it will be readily apparent to those skilled in the art having the benefit of this disclosure that any speech recognition tools, currently known or developed later, can be incorporated as a processing module into the 824 speech recognition engine.

In some embodiments of the invention, event extractor 825 comprises an event-based speech recognition module for building a weighted classifier scheme for use in speech recognition engine 824. In some embodiments of the invention, the pattern identifier 826 automatically identifies regions in a plurality of voice signals that contain event patterns. In some embodiments of the invention, the 827 unstressed region filter employed various techniques to filter out regions that are unlikely to result in robust unstressed detectors. In some embodiments of the invention, the boosted set simplifier 828 reduces the complexity of detector sets created through adaptive boosting algorithms. In some modalities. of the invention, event identifier 829 detects sequences of events instead of, or in addition to, detecting individual events. In some embodiments of the invention, alternate indicator detector 830 recognizes alternate voice indicators when aspects of the speech signal are corrupted. In some embodiments of the invention, the cascade detector array 831 automatically creates detector arrays. In some embodiments of the invention, the 832 speech generalizer improves generalization by maximizing the geometric margin, as explained above. In some embodiments of the invention, the perceptual cluster differentiation module 833 differentiates voice using perceptual clustering, as explained above. In accordance with these embodiments of the invention, the voice recognition engine 824 already provides voice data. In some embodiments of the invention, the recognized voice data is stored in one or more databases 834, wherein the one or more databases 834 is preferably coupled with a network 835. In some other embodiments of the invention , the recognized speech data is automatically sent to a short-term event memory 836 for speech-to-text processing.

In some embodiments of the invention, short-term event memory 836 is coupled with the plurality of text event processing modules to use the extracted events to provide a resulting text stream. In currently preferred embodiments of the invention, the event processing modules for text comprise a syllable head detector 837, a syllable categorizer 838, a syllable sequence perceptual indexing module 839, and a categorization module. subsyllable detail 840. Event stream processing modules for text provide a text stream with added prosodic information 841 embedded in it.

In some other embodiments of the invention, an apparatus is provided for extracting event data from a speech signal and detecting it.

word tion in it. Figure 8C illustrates a representation of a system 850 for event recognition and word detection that comprises event-based extraction and recognition of specific words. The 850 speech-to-text system includes an 852 acoustic analyzer 5 to receive an 851 input speech signal. The 852 acoustic analyzer is optionally coupled with a prosodic analyzer 853 and an event extractor 854. The event extractor 854 comprises a processing mechanism for automatically identifying regions in a plurality of speech signals that contain event patterns and extracting said events for word detection. Event extractor 854 is coupled with short-term event memory 855 to store the extracted voice events. Short-term event memory 855 is coupled with a plurality of word detection processing modules. In some embodiments of the invention, the word detection processing modules 215 comprise a syllable core detector 856 and a word detector 857. The word detection processing modules initiate a or more actions when a word is detected. The second processing module 862 comprises a pulsed neural network classifier. The information used for speech perception is not evenly distributed in frequency, amplitude and time. Temporal patterns are very important for speech recognition. Pulsed neural networks allow coding of voice information in temporal patterns of pulses and indistinct memory structures allow tolerance of temporal variability. The third processing module 863 comprises one or more consecutive speech recognition engines, as explained below. The 860 Voice to Alternative Text System also includes an 866 Acoustic Analyzer to analyze and digitize incoming voice signals.

867. The digitized voice signals are processed by one or more of the three processing modules 861, 862, or 863 and the results are fed to a decision module 868, which chooses the best recognized results and delivers an output of text 869.

Some embodiments of the invention involve segmenting a speech signal at perceptually important locations. This provides a means to extract not only perceptually relevant synchronizations, but also to synchronize the signal analysis with speech events, “5 thus avoiding all the problems of asynchronous fixed-frame analysis, as discussed above.

The method first performs a pre-segmentation filter using low-complexity filters that are based on certain aspects of human perception and the voice phenomena they are designed to detect. These filters detect the locations of perceptible patterns indicative of start, end, bursts, glottal pulses, and other significant speech signal events.

Pre-segmentation event filtering defines ranges that are used to synchronize certain category computations. The patterns of 2 15 features that have been extracted synchronously are further processed to create features over longer time scales and detect even higher levels of perceptual events such as phoneme boundaries, syllable head, etc.

Figure 9 illustrates an example of segmentations of a speech signal according to some embodiments of the invention. The voice signal in figure 9 contains the expression "Once". The signal changes shape several times over the course of the expression in ways that are visually apparent when viewing the waveform. The segmentations indicated by the short vertical marks at the bottom of the graph correspond to glottal pulse events during the "sound" part of the word.

The long vertical lines correspond to various types of voice sound threshold events. For reference, segment labels have been placed on the graph that indicates the phonetic identity of the segment. Signal conditions at transitions between phonemes vary by type of transition. At some thresholds the total energy changes abruptly, while at others spectral changes are associated with the event. Taken together, these various events allow feature extraction to be performed synchronously with speech events and provide perceptually relevant segmentation. : In some embodiments of the invention, signal segmentation is based on perceptual differences present in the speech signal.

Often, the information used for voice perception is not evenly distributed in time.

Human perception is sensitive to changes in stimuli.

In temporal signals such as voice, the time locations of significant changes (ie, events) provide the perceptual organization of the signal.

The relative timing of events and the characteristics of stimuli in their vicinity encode much of the perceptual information.

In general, perceptions of magnitude are not linear.

For example, it is known that the perception of sound intensity is logarithmic and commonly measured in decibels.

It can be shown that, for a wide range of perceptions, the barely perceptible difference in the stimulus is related to the original level of the stimulus.

However, this does not hold at the extremes and there is no perception at the low end until the stimulus level reaches a minimum level for neural activation.

At the upper end, once neurons begin to saturate, further increases in stimulus are not perceived.

In the operational range, for many types of stimuli, the change required for a perceptual response can be approximated by Weber's law: K = Al / lo; where lo is the original stimulus level, Al is the change in stimulus level, and K is an empirically determined constant that defines the threshold of the barely perceptible difference.

The right side of Weber's law formulation can be recognized as a contrast.

In the present invention, events are declared (ie, the detector trips) when the change in a relevant characteristic exceeds a perceptible threshold.

In the present invention, perceptual change is computed using a Weber's law-related perceptual contrast calculation.

Figure 10 illustrates a perceptual contrast formula used to compute perceptual change in accordance with some embodiments of the invention.

In this formula, the denominator of the right-hand relationship differs from the standard formulation of Weber's law in two ways: it includes the sum of the values that are contrasted, and it includes an additional factor e.

THE . factor and inhibits activation at very low levels to better simulate the perceptual response to very low-level stimuli.

It also makes the “5 formula numerically stable by avoiding division by zero when no stimulus is present.

Including the sum of the contrast values additionally flattens the perceptual contrast response at very low and very high levels.

For each perceptual characteristic measured (eg energy or frequency), appropriate values of € and perceptual thresholds are empirically established.

In some embodiments of the invention, a plurality of heterogeneous perceptual event detectors are created, each one based on some particular signal characteristic, measured on some particular time scale, and with its particular perceptual thresholds and thresholds. 215 The event detectors of the invention operate on various aspects of the signal at various scales.

First, pre-segmentation is performed by processing energy values through low-complexity filters that detect the temporal locations of bursts, closes and glottal pulses.

Feature extraction is then performed relative to pre-segmentation events.

Additional filters and detectors are applied to the synchronously extracted features to extract high-level features and events.

Additional Feature Processing and Extraction Techniques Segmented Circular Queue Memory Several event detector components involve comparisons of sums of feature values computed using multi-extent analysis windows, aligned in specific temporal relationships with one another. respect to the other.

To minimize the computational load of event detectors these sums are maintained using a segmented circular queue memory.

A circular queue is a first-in-first-out (FIFO) memory structure where new information is written to memory at |, the index of the oldest information in memory. After writing the new information into memory, the Index lo is advanced one module the size of memory (ie, the Index ly goes back to zero; when it reaches the end of memory). Running sums of values in memory can be maintained according to the process described below.

a: First, we initialize the circular queue memory locations, the running sum, and the index ly to zero. Then, in each step: the indexed value is subtracted from the running sum; the new value is added to the running sum; the new value is written into the circular queue and the Index ly is advanced by one module of the memory size.

The operation of a circular queue and its utility for efficiently switching running sums is illustrated in Figures 11A through 11C. Figure 11A illustrates a circular row memory in accordance with some embodiments of the invention. In Figure 11A, a 5-element circular row memory is represented at time "t" when a new value, "7", is about to be written. The new value will overwrite the oldest in memory which, in the illustrated example, has a value of "9". Before writing the new value, the sum of the values in the example memory is 25. Because the new value overwrites the oldest value, the current sum can be maintained by subtracting the old value and adding the new value. As can readily be seen, the computational complexity of maintaining running sums in this way is independent of the memory size. Only one subtraction and one addition are required regardless of memory size.

Figure 11B and Figure 11C illustrate an updated circular queue memory in accordance with some embodiments of the invention. More specifically, Figure 11B and Figure 11C show the update process continuing through the next two steps. To maintain running sums of values over various subsections of memory, the circular queue is segmented using additional indices, each of which has a fixed offset from index l. Each of the running sum subsections is maintained by simply subtracting the value that is about to be taken out of the subsection and adding the value that is about to become part of the subsection.

Figure 12 illustrates a segmented circular row for holding two running sums in accordance with some embodiments of the invention. THE . segmented circular queue is arranged to facilitate the maintenance of two running sums, one computed for the older half of the values in the “5 circular queue (i.e. subsection A) and the other computed for the newer half of the values in the circular queue. (ie subsection B). These sums are referred to as Z, and £g respectively. There is now a second index |; held at an offset equal to one-half the memory dimension from the lo index. At each step in time the value indexed by lp (that is, the oldest value in all memory) is subtracted from 3, and the value indexed by |; is added to Za, while the value indexed by |, is subtracted from Xg and the new value to be written to memory is added to ?g. The new value is written to the location in the lo index, and the | and |; they are then incremented by one module by the size of the memory. In the example . 15 now shown, the subsections of the memory are of equal size, form dismembered sets, and together cover the entire memory. None of these conditions are required by the method.

Figure 13 illustrates a segmented circular row according to some embodiments of the invention. In figure 13, subsection "A" is arranged so that it is completely within subsection "B". The total memory size as well as the dimensions of each subsection and the temporal arrangement of subsections are determined according to the purpose for which the sums are being kept. In some embodiments of the invention, circular rows are used to detect locations of abrupt changes. Several important voice events, such as beginning, ending, bursting stops, etc., are associated with abrupt semi-monotonic changes in the levels of some signal characteristics. A segmented circular row arranged generally as in Figure 13 can be employed to detect abrupt semi-monotonic changes. With the dimensions of subsections "A" and "B" properly determined, the perceptual difference between running sums of subsections "A" and "B" is computed at each time step. Times where the perceptual difference reaches a maximum and its magnitude exceeds its perceptual limit become candidates for segmentation points.

Additional 1 qualifications are applied to features that most closely mimic human perception by forcing a minimal time separation between detected “5” events.

Already at this stage, events can begin to be roughly classified, based on the direction of change in the event.

For example, events due to closures are distinguished from starts and bursts by the direction of energy change through transition.

In some other embodiments of the invention, circular queues are used to detect pulses and gaps in speech signals.

Some important voice events are associated with locations in time where some signal characteristics change abruptly for a very brief period of time and then return to a level similar to what they were at before the change.

If the brief change is to a greater value the change is called a "pulse". If the brief change is to a smaller value the change is called a "gap". A segmented circular row arranged generally as in Figure 5 may be employed to detect pulses and/or gaps.

With the dimensions of subsections "A" and "B" determined properly, impulses (gaps) are located when the average value in subsection "A" is above (below) the average value in subsection "B" by an amount perceptible adaptive limit.

As explained previously, the limit function is empirically determined.

The sizes of subsections "A" and "B" are determined according to the nature of human perception and temporal characteristics of the aspects of the signals to be detected.

Glottal Pulse Detection An important special case that illustrates the use of this approach is the detection of glottal pulse events.

Glottal pulse events are located using the following procedure.

First, the signal is bandpass filtered at the amplitude of the first formant.

Then Teager energy is computed as Teager (t) = x(t) * x(t) — x(t-1) * x(t+1); where x(t) is the input value at time t.

Being a function of amplitude and frequency, Teager energy emphasizes the pulse locations of glottal pulses, which are associated with local maxima of the energy and high frequency components. ' Finally, the signal is segmented using an impulse detector arranged generally as in figure 13. The detector is based on running sums of absolute values of Teager energy. In the preferred mode, the dimensions of subsections "A" and "B" are set to 2 ms and 10 ms respectively. The detector is in a high state whenever the average Teager energy in subsection "A" is greater than the perceptual threshold K multiplied by the average Teager energy in subsection "B". The value of K was chosen to be 1.3. The dimensions of subsections "A" and "B", and the value of the K multiplier have been found to be useful for detecting glottal pulse locations. Values other than those described herein may be used within the scope of this invention.

The glottal pulse detector just described creates two events of — 15 locations for each glottal pulse, one at the rising edge of the pulse and one at the falling edge of the pulse. The drift period is defined as the period between two sequential rising threshold events. The pulse duration is estimated by the time between the rising threshold and the subsequent falling threshold. The relationship of pulse duration to total delay period is related to the "open quotient", a characteristic of voiced voice that can be useful in some speech processing applications. Furthermore, during the open part of the retraction period the subglottal cavities are acoustically coupled with the oral cavities creating slightly different formant patterns during this part compared to the closed part patterns. This fact can be exploited advantageously by arranging feature extraction in relation to these events.

Figure 14 illustrates a representation of a glottal pulse detector output in a voiced voice segment according to some embodiments of the invention. In figure 14, the output of the glottal pulse detector divides the signal into "high" and "low" segments. The segments represent times when a relevant characteristic (in this case Teager energy) is perceptually above the norm. This arrangement creates a segment for the duration of the pulse or gap.

For some applications it may be preferable to mark a pulse or gap rather than a segment.

In these cases the selection of specific event times can be determined by one of several alternative methods which include, but are not limited to: e —selecting the midpoint between the ascending (descending) and descending (rising) limits ; and —select ascending segment limit; and —select descending boundary of the segment; * —select the maximum (minimum) characteristic value within the segment; and — select the point of maximum perceptual contrast within the segment.

Glottal pulse detection as outlined above is based on detection when the average value of a certain signal characteristic (e.g. Teager energy) within a centrally arranged window deviates significantly from the average of the same characteristic calculated over a period of time. - longer time odo.

Segmented circular rows arranged in general as in Figure 13 can be used to segment any modulation signal by identifying regions where a selected voice characteristic (eg energy or formant frequency) perceptually deviates from its long-term norm.

Because the computational cost of maintaining the running sums used by the detectors is independent of the size of the subsections, they can be used to segment large scale modulations as well as short pulses.

Syllable Nucleus Detection To illustrate this point, a syllable nucleus detector was constructed using a segmented circular row arranged in general as in figure 13, to maintain running sums of Teager energy, computed exactly as for the glottal pulse detector except the dimension of subsection "A" which was set to 60 ms and the dimension of subsection "B" which was set to 100 ms.

Figure 15 illustrates a representation of an output waveform according to some embodiments of the invention. Figure 15 shows the waveform and detector outputs for the word "Once" spoken twice: the first time normally and the second time in a whisper. As can be seen, this detector usually groups the central parts of syllables.

'5 Some embodiments of the invention involve methods for recognizing speech patterns using formant extraction. As the voice is produced, the configurations of the organs of articulation (eg, tongue, jaw, lips) create dynamic patterns of resonances and anti-resonances in frequency spectra called formants. During the sonorous voice, the sound is generated both by diffuse "air noises" and by a strongly organized harmonic structure. Both the diffuse and harmonic components contribute to voice understanding and both are variably invoked under different noise conditions. Diffuse "air noises" interact with and are shaped by formants, revealing them to be relatively attenuated. Strong resolved harmonics create relatively sharp peaks in the spectrum and, if not processed properly, make it difficult to precisely locate nearby formants. Harmonic series provide an excellent means of determining offset, even when the offset period frequency itself is missing from the signal. Experiments have shown that amplitude modulated harmonics can be used to recreate intelligible speech that "ignores" noise. During voiceless speech, temporally perceptible changes divide the signal into semi-homogeneous segments.

Formant Extraction In some embodiments of the invention, a formant extraction process is performed as described in Figure 16. Figure 16 illustrates a workflow 1600 for performing formant extraction in accordance with some embodiments of the invention.

Workflow 1600 begins when the segment samples are fitted to the Hamming window 1601 with a window span equal to the span of the segment, where the segment corresponds to a period of offset during the voiced voice. The windowed samples are then processed through a bank of 1602 broadband filters. In some embodiments, the bandpass filters have bandwidths of 400 Hz and are spaced at 50 Hz centers covering the width of 400 Hz. 450 Hz up to 4000 Hz. Next, the workflow computes the instantaneous amplitude and frequency of each filter is computed using the DESA-1 16 03 technique. Based on their numerical qualities, the computed values are judged as "valid" or "not valid" in step

1604. It then counts and stores "valid" estimates in a temporary memory. Next, a histogram whose boxes represent frequency amplitudes is initialized 1606, where for each valid estimate, the histogram box representing the estimated instantaneous frequency is incremented by the logarithm of the corresponding compressed estimated instantaneous amplitude F. Then the peaks of the attenuated histogram are selected as 1607 formant candidates, the formant frequencies, bandwidths (sigmas), and amplitudes are retained as 1608 features, and the delta features are computed on the formant tracks by adjusting - linear ment 1609. Finally, at locations of noticeable change in formant patterns, 1610 events are generated. 12th Octave Filterbank Processing In some other embodiments of the invention, a 12th octave filterbank processing process is performed in the segmented signal using narrow pass bands at lower frequencies and wider pass bands at higher frequencies mimicking the frequency resolution trends found in human hearing. Fig. 17 illustrates a workflow 1700 for performing formant extraction in accordance with some embodiments of the invention. Workflow 1700 starts when the segment samples are synchronous with the signal, fitted to the Hamming window 1701 with a window span equal to the span of the segment, where the segment corresponds to a period of one offset. Then, the window-fitted samples are processed through a bank of spaced filters.

12th octave 1702 and the instantaneous amplitude and frequency of each filter are computed using the DESA-1 technique 1703. Based on their numerical qualities, the conjugate values are judged 1704 "valid" or À "not valid", where "valid" estimates are counted and stored in a temporary memory for the range 1705.

Next, a histogram is constructed, whose boxes correspond to the central frequencies of each filter in the 12th octave filter bank 1706, in which for each estimated value, the histogram box whose amplitude includes the estimated instantaneous frequency is incremented by the logarithm of the compressed estimated instantaneous amplitude. Then, the histogram weights are multiplied by a weighting function based on the sensitivity of the ear at different 1707 frequencies. After computing the histograms, the histogram box energy patterns are summed SÁ in harmonic combinations to detect the strongest harmonic sequence. - — 15 tecoma strongest energy 1708, where the fundamental of the harmonic sequence is used as a distance estimate. If the application requires even more accurate estimates, narrow bandpass filters are centered on the estimated harmonic frequencies and recomputed 1709. This process quickly converges to highly accurate estimates. Finally, the ratio of harmonic energy to total energy is computed as a 1710 loudness measure, where harmonics amplitude ratio patterns are maintained as characteristics, where the ratio is used in automatic speech recognition. .

Use of Offset Periods In some embodiments of the invention, the beginnings and offsets of harmonic trails can be determined by relative amplitudes of offset period to offset period. Abrupt changes in the amplitude of harmonic tracks are associated with the interaction of harmonics with the formants, and abrupt changes indicate a change in interaction, which may be due to a change in distance or a change in formant. These changes are indicative of a transition location. Events can be generated in response to these changes using the previously exposed filter methods.

It should be noted that these events, when they occur, will be synchronous with the synchronism of the glottal pulses.

Vocal Tract Normalization and Attenuated Phoneme Segment Recognition In some embodiments of the invention a process of vocal tract normalization and attenuated phoneme segment recognition is employed to resolve complications inherent in using formant patterns as features.

The formant patterns generated by a speaker simultaneously encode information about the voice sounds that are produced and the length of the speaker's vocal tract.

This complicates the use of format standards as characteristics.

It was noted in Watanabe, et al., Reliable methods for estimating relative vocal tract lengths formant trajectories of common words, : 15 IEEE transactions on audio, speech, and language processing, 2006, vol. 14 pp. 1193 to 1204, that the formants for two speakers producing the same voice sound have a relationship inversely proportional to the relationship of their vocal tract extensions: La/Lg = Fna/FnB As different voice sounds are produced, the vocal tract extension of the speaker is continually modified through the dynamic reconfiguration of the organs of articulation.

For a given speaker, as each sound is produced the formants will move up or down because they are changing the length of the vocal tract.

Applying Watanabe's formula to the formant pattern of speaker "A" pronouncing a certain voice sound, and the formant pattern of speaker "B" pronouncing the same sound, gives an estimate of their relative vocal tract spans for each measured formant.

Some aspects of the invention are based on the following information.

First, if a speaker "A" and speaker "B" are producing the same sound, the relative vocal tract estimates based on each of the various measured formants will approximate the true value and therefore one will be similar to the other.

Next, if speaker "A" and speaker "B" are producing different sounds, vocal tract extension estimates based on

from one of several measured formants will be divergent. Additionally, if the transition of a certain voice sound involves lengthening (shortening) the length of the vocal tract when spoken by speaker "A", it will also involve lengthening (shortening) the length of the vocal tract of speaker "B" but by different amounts based on your physiology.

In some embodiments, formant values for each voice sound, as spoken by a reference speaker, are recorded. Reference speaker formant measurements can be based on a single speaker or more, preferably taken as the average of measurements from many speakers. At recognition time, each segment is processed to produce formant values as described previously. Each voice sound (i.e., phoneme or partial phoneme) is in turn assumed to be what is being spoken, and the formant values of the current i segment are used to compute estimates of the relative vocal tract length of the voice. current speaker to that of the reference speaker. Based on the consistency list, the relative probability of each voice sound can be established. As the voice trajectory approaches the target configuration of each approved formant pattern, the consistency of the estimates will increase and these target times will tend to be higher for the perceived voice sound. The confidence that can be applied to such perceptions is dependent on the sound conditions of voice and noise. When voice sounds are determined with high confidence, they become useful signal reference points for constraining possible patterns in regions with less confidence. Consecutive Parallel Automatic Speech Recognition Engines Some embodiments of the invention involve using a plurality of consecutive parallel automatic speech recognition (ASR) engines in burst mode temporarily overlapping each other to reduce latency and improve accuracy. Each ASR engine can be of similar or different design and origin, but all must be able to produce acceptable results in the target language in the core part of the segment within the minimum segmentation time frame. The results of consecutive processors are analyzed by weighting the words produced during the middle part of each segment larger than the words produced at the beginning and end, synchronizing the segments for the best fit, and the words with the highest weight are selected for output. These modalities involve the use of multiple ASR mechanisms on overlapping audio voice segments to reduce latency and improve accuracy. Consecutive parallel approach increases accuracy while reducing latency.

For example, if an ASR arbitrarily segments an input speech signal at x seconds, the output tends to be more accurate at location x/2, and less accurate at the beginning and end of the segment, since the context highest in both forward and backward directions is found AND in the central location. Given this observed behavior, one might be able to leverage this information simply by executing n instances of an ASR mechanism in batch mode, segmenting the input signal into bursts of x seconds that overlap by x/n seconds, and alternating the forwarding of these segments between each mechanism. If n = 2, at the same time that engine B is working on recognizing its segment, the output of engine A is analyzed together with the previously output word stream to statistically reinforce, correct and provide the words from engine A. Then, at the boundary n of the second input, the output parser and processing tasks switch obligations between the engines.

Looking at a typical ASR engine useful in a consecutive setup, it can be seen that x seems to work best when set to around three seconds when using a three thousand word WSJ English language model. This allows the possibility to use the engine, which is designed and optimized to work on long expressions, to be adapted for use in environments where low latency is required.

In other words, if x = 3, the first voice segment in 0.0 to 3.0 seconds will be presented for transformation to engine A.

Segment 1.5 to 4.5 will then be presented for engine B, etc.

Figure 18 illustrates a representation of two consecutive processing mechanisms, overlapping in time, operating in a sequence of expressions according to some embodiments of the invention.

As shown in Figure 18, the words, "is falling from the sky" are provided by mechanism A, and "done the sky today at" comes from mechanism B. Employing statistical methods discounting the weight for each word at the ends of each segment that takes into account the reliability factor for those words, one can end up with an apparent continuous string of words such as "is falling from the sky today at" with a fixed latency of 3 seconds.

The weighting analysis and output mechanism may include one or more algorithms in the following categories as well as others to determine which words will be added to the final output string. For example, an algorithm may involve simple weighting of the central words in a segment with values greater than the words at the limits of the segment, acoustic and prosodic indicators gained from the original speech signal, statistical analysis of the words to be provided for reinforcing most likely output weights, grammar rules for selecting the most likely output, or other statistical and machine learning methods. Automatic Punctuation Some embodiments of the invention involve automatic insertion of punctuation marks in unpunctuated text. An automatic punctuation system is a system that inserts punctuation marks (periods, commas, question marks, exclamation marks, apostrophes, quotation marks, parentheses, ellipses, semicolons, and colons) into unpunctuated text.

Figure 19 illustrates a speech-to-text system 1900 that includes an automatic scorer in accordance with some embodiments of the invention.

In some embodiments of the invention, unpunctuated text may give rise to text 1901, or voiced language 1902 which is then transcribed to text by an automatic speech recognition system 1903.

The transcribed text or native 1901 text is sent to the 1905 auto punctuation. The 1905 auto punctuation creates text that is more easily readable and less ambiguous due to the correct placement of punctuation marks.

In some embodiments of the invention, the autoscorer 1905 is coupled with a database 1904 that contains training data. The auto scorer uses one or more Baysian algorithms that are trained on a large amount of training texts that are scored correctly. The scoring patterns in the training data are analyzed to create a set of rules that describe the scoring patterns in the text.

Once the punctuation has been trained on a sufficient amount of text, its rules can then be applied to new text to predict where punctuation marks should be inserted.

In some embodiments of the invention the autoscorer 1905 comprises a plurality of processing modules. As shown, the autoscorer includes a first statistical processor 1906, a second statistical processor 1907, and a third statistical processor 1908.

In some embodiments, the first 1906 statistical processor identifies places where scores should be entered based on statistical rules. A training process is conducted to develop the rules. The training process involves analyzing the correlations between specific words and punctuation marks in a large amount of correctly punctuated texts. The rule set is derived from this analysis. The ruleset can then be applied to new unpunctuated text to predict likely locations for punctuation marks. The output of this process is a series of opinions about where the punctuation marks should be inserted. | In some embodiments, the second 1907 statistical processor trains on the correlations of word classes with punctuation marks. This process is based on a word class marker that analyzes the sentence structure in the training data and assigns a word class label to each word.

Examples of word class labels are noun, verb, adjective, preposition, etc.

The process then builds a set of rules based on its observations of how certain word classes correlate with punctuation marks.

Then the rule set can be applied to new text.

The output of this process is a series of opinions on where to place punctuation marks within the text.

In some embodiments, the 1908 third statistical processor uses weighting based on average sentence lengths.

The third component of the statistical scorer is based on the number of words that typically form sentences in a particular text.

As with the other processes, it trains on a large amount of punctuated text Í correctly.

Rules are developed based on the amount of n- , 15 grams that occur in text units that are limited by punctuation.

In some embodiments of the invention, the results of the first statistical processor 1906 and the second statistical processor 1907 are two sets of opinions as to where punctuation should be inserted in a text.

The results of the third 1908 statistical processor are then used as a kind of tiebreaker to resolve situations when decisions are in conflict.

For example, if the first stat processor 1906 predicts that a dot is needed after the fifth word in a string of words, and the second stat processor 1907 predicts that a dot is needed after the third word, the results of the third stat processor 1908 must be called upon to make the decision, as both are unlikely to be correct, as a two-word sentence would be formed.

In some embodiments, the third statistical processor 1908 assigns greater weight to the results of either the first statistical processor 1906 or the second statistical processor 1907 based on its knowledge of the typical sentence length in this type of document.

If sentences in the document type are typically very short, the third statistical processor 1908 should assign greater weight to the output of the second statistical processor 1907. If, on the other hand, sentences in the document type are usually five words or longer, he should give more weight to the opinion generated by the first 1906 statistical processor.

Once the decision-making step is complete, the result is passed to a 1909 decision module that will make the final decision on where to enter scores, together with information from a 1910 rules-based scoring module and a 1910 rule-based scoring module. of leave/pause

1911. In some embodiments, a 1910 rule-based punctuation module uses a set of rules on linguistic structure to determine where punctuation marks should be inserted in the text. The 1910 rule-based scoring module 2, 15 is coupled with a 1916 lexical database.

The 1910 rule-based punctuation module can identify several functional classes of words, including subject pronouns, object pronouns, relative pronouns, modals, conjunctions, definite articles, dates, and certain categories of verbs. In some embodiments, the 1916 lexical database includes word class information.

Once the program has identified a member of one of the functional categories, it proceeds to search for the next context, looking in a text window consisting of the identified item and two preceding and following words. Specific word categories or word classes that occur in the context window will indicate the need for a comma at some point in the word sequence. Linguistic rules serve as an instruction list for where commas should be inserted. As an example, when the program identifies a subject pronoun (I, he, she, we, they) it checks the context window for the occurrence of other categories. If, for example, the subject pronoun is preceded by an adverb or a participle (with certain verb participles expected) the program will predict that there must be a comma after the word that precedes the identified word.

The rule-based scorer can process a string of text words or a pre-existing text file.

The output of the rule-based scorer is a series of opinions about where to insert commas.

In some embodiments, the 1911 Away/Pause module is different from other components in that its input is an audio file that contains human voice.

The other components operate on text, although the text may have originated as audio data which was then transcribed.

The 1911 withdrawal/pause module operates on the observation that in the human voice, significant withdrawal changes that occur over a short period of time and are correlated with a period of silence are usually indicative of a need for punctuation.

For example, if a given point in an audio file shows an abrupt drop in distance (30% or more) occurring in a short period of time (275 ms), that is a likely indicator that the speaker has encountered the end of a sentence.

The presence of a pause following this pattern tends to confirm that a location for a punctuation mark has been identified.

The away/pause scorer tracks the distance of an audio file and signals when the correct conditions have been met to indicate scoring.

The leave/pause scorer provides opinions on where punctuation marks should be inserted.

In some embodiments, decision module 1909 receives input from auto scorer 1905, rule-based scorer 1910, and skip/pause module 1911. Based on known characteristics of the text type, decision module 1909 assigns higher or lower weights for each of these results to make a final determination as to whether or not a punctuation should be inserted at a given point in the text.

Claims (20)

U7 CLAIMS 1. Speech recognition mechanism, comprising: an acoustic analyzer for receiving and digitizing a speech encoding signal; an event extractor for extracting events from said speech signal, wherein said events or patterns of events are highly relevant in speech recognition; and a speech recognition module coupled to said event extractor, wherein said speech recognition module uses events to initiate at least one action in response to detected content. 2. Speech recognition engine according to claim 1, wherein said initiated action is the conversion of speech content of said signal into at least one text string. [ 3. Speech recognition engine, according to claim1, wherein said action initiated is to suppress audio output from a system when certain words are detected. 4. Voice recognition mechanism, according to claim 1, wherein said initiated action is to respond to detected commands. 5. Speech recognition engine, according to claim 1, wherein said event extractor additionally comprises: a training corpus of digitized voice dictions of known classes; a plurality of stress detectors, each stress detector comprising a method for determining the presence of an event; and a means for mounting an array detector, wherein said array detector comprises a group of unstressed detectors which acting together are better at determining the presence of an event than any of the constituent unstressed detectors. 6. Speech recognition engine according to claim 5, wherein said array detector is iteratively created using being a reinforcement algorithm, thereby forming a reinforced array detector. 7. Speech recognition mechanism according to claim 6, wherein said event extractor includes means for simplifying said enhanced array detector, thereby forming a simplified array detector. 8. Speech recognition engine according to claim 7, wherein said event extractor includes means for automatically converting said simplified array detector into a cascade detector. 9. Speech recognition mechanism according to claim 1, wherein said event extractor additionally comprises a means for classifying voice into perceptual groups and disambiguating i between alternative perceptions. 215 10. Speech recognition mechanism according to claim 1, wherein said event extractor additionally comprises a filter for rejecting regions of a digitized speech signal that do not contain events and that are not likely to result in an event. robust detectors. 11. Speech recognition engine according to claim 1, wherein said event extractor further comprises a sequence of events recognizer, wherein said sequence of events recognizer detects sequences of events. 12. Speech recognition engine according to claim 1, wherein said event extractor additionally comprises an alternative interpretation detector configured to recognize alternative speech interpretations to strengthen recognition, for example, when aspects of the voice signal are corrupted. 13. A speech recognition engine according to claim 1, further comprising a signal synchronization mechanism, comprising: a pre-segmentation filter for defining intervals that are used to synchronize computations of characteristics; a means for segmenting said digitized signal based on perceptible gap differences, thereby forming a segmented signal; and a feature extractor for extracting event-related features from said segmented signal. 14. Speech recognition engine according to claim 1, further comprising an automatic punctuation for automatically inserting punctuation into at least one text string. 15. Voice recognition method, which comprises: establishing a training of unstressed classifiers based on training examples; build a set of detectors; : receive a voice signal; 15 digitizing said voice signal; processing said speech signal using said set of unstressed detectors, wherein thereby recognizing the presence of at least one event, wherein said event comprises a pattern in said speech signal that is highly relevant in the voice recognition; and processing said events to recognize voice. 16. The voice recognition method according to claim 15, wherein the step of building a set of detectors comprises the steps of: storing a plurality of voice signals, wherein said voice signals voice comprise training examples stored in an automatic voice recognition system; extracting event patterns from said plurality of speech signals, wherein said patterns comprise distinct characteristic locations in said speech signals; accessing a sample of said plurality of speech signals having corresponding event patterns; align individual voice signal events across samples, wherein said alignment comprises aligning said events of said individual speech signals based temporarily on said corresponding event patterns; optionally adjusting said individual signals to a common time duration; evaluating the effectiveness of a plurality of unstressed detectors in detecting said event patterns; applying a weighting scheme to said plurality of unstressed detectors, in which the most effective unstressed detectors receive the highest weights; adding at least one stress detector to said plurality of stress detectors; and iterating said steps of: [accessing a sample of said plurality of voice signals that; 15 has matching event patterns; align individual speech signal events across samples; optionally adjusting said individual signals to a common time duration; evaluating the effectiveness of a plurality of unstressed detectors in detecting said event patterns; applying a weighting scheme to said plurality of unstressed detectors based on the relative effectiveness of said unstressed detectors; and adding at least one additional stress detector to said plurality of stress detectors; wherein said iteration step is executed until said effectiveness of said weighting scheme executes an efficiency pattern set to detect said event patterns. The method of claim 16, wherein the step of accessing a sample of said plurality of speech signals that have corresponding event patterns further comprises automatically identifying regions in said plurality of speech signals that contain include said event patterns, comprising the steps of: aligning said plurality of speech signals relative to a common time axis; optionally adjusting the durations of each individual speech signal from the plurality of speech signals to one; projecting syllable centers and/or other event locations of said individual voice signals within the same time axis in the form of projected syllable centers and projected event locations; and identifying regions on said time axis that have a concentration of syllable centers or other event locations in the form of regions in said plurality of speech signals that contain said event patterns. The method of claim 15, wherein the step of accessing a sample of said plurality of speech signals having corresponding event patterns further comprises automatically identifying regions in said plurality of speech signals that contain said event patterns, which comprises the steps of: accessing a training set; convert said voice signal into regions of time-path space that contain all the events of the positive training examples; compute negative example counters for all regions; select a region of time-trajectory space with fewer events from the negative training examples; eliminate negative examples with no events in the selected region from further consideration; and repeat the steps of: computing the negative examples counters from the remaining negative examples in each region; select a region of time-trajectory space with fewer events from the negative training examples; and eliminate negative examples without events in the additional consideration of selected region shape; until a cascade is created that operates perfectly on said training set. 19. A method of operating two or more recognition systems in series, wherein said two or more voice recognition systems perform detection and analysis of a voice signal at overlapping time intervals, said method comprising : configure the time intervals to be used in each speech recognition engine, where said intervals are reconfigurable; configuring the overlap of intervals, wherein the overlap is reconfigurable, and wherein the overlap is adjusted to reflect the most information-rich parts 1 of said speech signal; 215 routing said detection and analysis between said speech recognition mechanisms; weighting the results of said speech recognition mechanisms, in which a higher weight is given to the results taken from the middle of the range, and generating at least two opinions as to the identity of the words within a single time interval; and determining which opinion of the at least two opinions best estimates a representation of said speech signal. 20. Speech recognition mechanism, comprising: an acoustic analyzer for receiving and digitizing a speech signal —in the form of a digital diction; a speech recognition module coupled to said acoustic analyzer, wherein said speech recognition module converts said digital diction into at least one text string; an automatic scoring mechanism coupled with a database containing training data, wherein said automatic scoring mechanism includes at least one statistical processor for adding the score to said text string using said data ' statistic-based punctuated text training TM; a rule-based scorer coupled with a lexical rules database, wherein said rule-based scorer adds punctuation to said text string using rules from said lexical rules database in the form of rule-based punctuated text ; and a decision module for determining whether said scored text or said statistically based scored text produces a better scored result.