WO2014128360A1 - Synchronisation de contenu audio et vidéo - Google Patents

Synchronisation de contenu audio et vidéo Download PDF

Info

Publication number
WO2014128360A1
WO2014128360A1 PCT/FI2014/050138 FI2014050138W WO2014128360A1 WO 2014128360 A1 WO2014128360 A1 WO 2014128360A1 FI 2014050138 W FI2014050138 W FI 2014050138W WO 2014128360 A1 WO2014128360 A1 WO 2014128360A1
Authority
WO
WIPO (PCT)
Prior art keywords
audio
video
frames
data frames
synchronization
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/FI2014/050138
Other languages
English (en)
Inventor
Jukka Ojanen
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LINKOTEC Oy
Original Assignee
LINKOTEC Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by LINKOTEC Oy filed Critical LINKOTEC Oy
Publication of WO2014128360A1 publication Critical patent/WO2014128360A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8106—Monomedia components thereof involving special audio data, e.g. different tracks for different languages
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H04N21/43072—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen of multiple content streams on the same device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4341—Demultiplexing of audio and video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams

Definitions

  • the present invention generally relates to multimedia playback (reproduction) and particularly to methods, apparatuses and software for synchroniza- tion between audio and video content obtained from different sources.
  • a dedicated DVD or Blu-Ray player, or a computer program decoding and reproducing video and audio content from a DVD or Blu-Ray disc can play back discrete multi-channel audio and video content.
  • multi- channel audio content means more than the left and right channels commonly associated with stereo audio.
  • notions like m.n. eg 5.0, 5.1, 7.1, etc. are used to denote the number of audio channels, the integer part denoting the number of audio channels with full frequency range and the decimal part (usually zero or one) denoting the number subwoofer channels with the frequency range limited to frequencies whose contribution to spatial sensation is small and which are not easily reproducible by small speakers.
  • Discrete multi-channel audio means a coding/decoding scheme wherein the m or m+n channels are decoded into separate information streams, as opposed to, say, decoding schemes wherein the front left and right channels are amplitude-encoded into separate information streams, while the rear left and right channels are phase-encoded into the information streams whose amplitude modulation carries the front audio channels.
  • lip synchronization (International Telecommunication Union) terminology database defines lip synchronization (“lip-sync") as follows: "Operation to provide the feeling that the speaking motion of the displayed person is synchronized with that person's voice. Minimization of the relative delay between the visual display of a person speaking and the audio of the voice of the person speaking. The objective is to achieve a natural relationship between the visual image and the aural message for the viewer/listener.”
  • the characterization of sensitivity to the alignment of sound and picture includes early work at Bell Laboratories.
  • ITU-R Recommendation BR.265-9 (and its earlier versions) states that the accuracy of the location of the sound record and corresponding picture should be within ⁇ 0.5 film frames, or about ⁇ 22ms.
  • ITU-R published BT.1359 recommendation, which states that the threshold of detection ability of lip sync errors is about +45ms to -125ms (+: audio early, -: audio late) and that the threshold of acceptability is about +90ms to -185ms, on the average.
  • the ITU recommends that the timing difference in the path from the output of the final program source selection ele- ment to the input to the transmitter for emission should be kept within the values +22.5ms and -30ms.
  • EBU Technical Recommendation R37-2007 gives a range of +40ms and - 60ms at the output intended to feed broadcasting transmitters.
  • the present disclosure acknowledges the fact that there are several sources of uncertainty to gain accurate synchronization of audio to video. Operation of synchronization is dependent on the devices environments in which the devices are used.
  • One of the problems associated with the above arrangement is that in connection with video or multimedia playback from an external source, current internet technologies do not support discrete multichannel audio, wherein multi- channel means more than the two channels of a stereo system.
  • a related problem is that the external source may not offer audio in a language desired by the user of the client terminal. If the original audio source is replaced by another audio source, the need for accurate synchronization in a data processing device emerges as another problem. Accurate synchronization is problematic in general-purpose data processing devices wherein multiple processes compete for processing resources, which may lead to variations in execution times.
  • An object of the present invention is thus to eliminate or mitigate at least one of the problems identified above. This object is attained by aspects of the inventions as defined in the attached independent claims.
  • the dependent claims and the following detailed description and drawings relate to specific embodiments which solve additional problems and/or provide additional benefits.
  • An aspect of the invention is a method for synchronizing audio and video in a data processing device.
  • the method comprises a number of acts performed by the data processing device. These acts are labeled here for the purpose of discussion only, and such labeling does not restrict execution of the steps, unless execution of a step clearly requires an outcome of an earlier step:
  • processing of the received video content and the first audio content comprises a first synchronization of the first audio content with the video content, and based on the first synchronization, outputting video data frames associated with a first time code information.
  • the method further comprises the following acts performed by a HTTP Server Application in the data processing device:
  • the acts performed by the HTTP Server Application in the data processing device further comprise a second synchronization of the PCM audio frames with the video data frames, wherein the second synchronization comprises:
  • An illustrative but non-exhaustive list of representative data processing devices includes a personal computer, a laptop computer, a palmtop computer or a tablet computer, a smart phone, a media player with online connections, or the like.
  • the data processing device comprises a video or multimedia player applica- tion which is capable of acting as a HTTP client. Apart from the ability of acting as a HTTP client, steps a) and b) represent conventional functionality of a virtually any video or multimedia player application.
  • the video content may be obtained from a source internal to the data processing device or external to it. Examples of internal sources include optical disks residing in an optical disk drive and locally stored multimedia files, such as those stored in MP4, DivX, or comparable formats. Examples of external sources include online repositories accessed via content servers.
  • an HTTP server application in the data processing device offers an application programming interface ["API"] to the video or multimedia player application, which acts as the HTTP client.
  • the HTTP server application in the data processing device performs steps labeled c) through i).
  • Step d) comprises receiving audio content from an audio source, wherein the received audio content comprises audio data samples which are associated with audio-related time code information, which in the present context will be called a second time code information.
  • the audio data samples may comprise a file or stream of PCM frames or other time-related samples
  • the second time code information comprises a frame or sample number and/or a time stamp from the beginning of the file or stream or a portion of it, such as a track.
  • Step e) comprises producing pulse-code modulated ["PCM"] audio frames based on the received audio data samples.
  • the HTTP server application acts as an audio player application. In itself, outputting the audio samples is normal functionality of an audio player.
  • a problem underlying the invention is that without the additional steps described later, the outputted video and audio are likely to fall out of synchronization with each other. Accordingly, the acts per- formed by the HTTP Server Application in the data processing device further comprise a second synchronization of the PCM audio frames with the video data frames, wherein the second synchronization comprises steps f) through i).
  • the HTTP server application offers an Application Programming Interface ["API"] to the multimedia or video player acting as a HTTP client and receives the first time code information or its derivative from the HTTP client over the API.
  • API Application Programming Interface
  • the HTTP server application acting as an audio player application doesn't need or use the video content processed by the video player / HTTP client
  • the HTTP server application needs the audio-related first time code information or a derivative of it at least until the second synchronization has been ac- complished, after which the audio content outputted by the video player or the audio-related first time code may optionally be muted.
  • the video- related first time code information may be a frame number from the start of the file or stream
  • a derivative may comprise a modulo function (a number of least significant bits) of the frame number or a time stamp derived from the frame number.
  • the act of sending the first time code information by the video or multimedia player application on one hand, and the act of receiving the first time code information by the HTTP server application acting as the audio player application on the other hand may be accomplished by techniques used in cli- ent/server architectures.
  • a software company providing the HTTP server application / audio player application may also publish an applet, such as a length of JavaScript code whose execution by the web-enabled video or multimedia player application causes the latter to output the video-related first time code information over the API to the HTTP server application / audio player applica- tion.
  • Step g) comprises repeatedly determining a time difference between the second time code information and the first time code information respectively associated with concurrently outputted audio data samples and video data samples.
  • the HTTP server application compares 1) the second time code information associated with outputted PCM audio data frames with 2) the first time code information associated with video data samples outputted concurrently with the audio data samples.
  • such comparison indicates whether video or audio or neither is leading or trailing the other.
  • it is not very useful to jump to conclusions based on individual time differences. For instance, the time differences may be spurious and disappear spontaneously. Furthermore, overly rapid corrective measures may be more disturbing than periods of failed synchronization between the audio and video outputs.
  • step h) comprises calculating at least one moving average function based on the repeatedly determined time difference.
  • a "moving average function” means a function whose result has at least 0.5 correlation with a mathematically precise definition of moving average.
  • Reasons for deviating from a mathematically precise moving average may include a desire to reduce processing load by substituting coarse table lookup values for accurate multiplication and division results, for example.
  • step i) the moving average function(s) is/are used to indicate if the outputted audio data samples lead or trail the outputted video data samples, or are synchronous with them. If the moving average function (s) indicate that the outputted audio data samples lead the outputted video data samples, a corrective measure is taken by oversampling the received audio data samples. Oversampling means that a set of audio samples is processed to yield a set of processed audio samples whose combined length is longer than the nominal lengths of the audio samples. Conversely, if the moving average function(s) indicate that the outputted audio data samples trail the outputted video data samples, a corrective measure is taken by undersampling the received audio data samples. Undersampling means that a set of audio samples is processed to yield a set of processed audio samples whose combined length is shorter than the nominal lengths of the audio samples.
  • a simple implementation of the invention may utilize one moving average, while more ambitious implementations may involve more, such as three, moving averages, each moving average calculated over a respective plurality of audio or video samples. For instance, assuming that a "short" moving average, ie, a moving average calculated over a small number of samples, indicates that audio output is leading or trailing video output, a relative small corrective measure may be taken. In contrast, if a "long" moving average, ie, a moving average calculated over a large number of samples, indicates that audio output is leading or trailing video output, a larger corrective measure may be taken.
  • the inventor has discovered that an implementation of the inventive audio replacement and synchronization scheme as an HTTP server has the broadest support across different platforms.
  • the HTTP server typically resides in the same data processing device, such as a mobile terminal, that also hosts the client component.
  • Figure 2 is a flow chart illustrating an operating principle according to an embod- iment of the invention
  • Figure 3 is a block diagram illustrating an exemplary hardware implementation
  • FIG. 4 is another block diagram showing major hardware blocks in a data processing device.
  • Figure 1 shows an exemplary network environment in which the invention and its embodiments can be used.
  • a client terminal CT accesses a video repository VR.
  • the term video repository VR does not exclude audio, and a typical video repository VR indeed stores audio content as well.
  • the term video repository VR implies that the invention and its embodiments enable use of video content from the video repository VR and audio from an audio repository AR, which is or may be a repository entirely distinct from the video repository.
  • the video repository VR is a remote repository, typically implemented as a data base server, which the client terminal CT accesses over a data network DN and, assuming that the client terminal CT is a mobile terminal, over an access network AN.
  • the data network DN is the internet or its closed subnetworks, commonly called intranets or extranets.
  • the access network AN may be a cellular mobile network or a wireless local-area network (WLAN), to name just a few of the available options.
  • the video repository VR is a local memory resource in the client terminal CT, such as a local DVD or Blu-Ray disc or a multimedia file, such as an MP4 or DivX file, stored locally in the client terminal.
  • the client terminal CT comprises means for video or multimedia playback.
  • video or multimedia playback was typically implemented via dedicated playback circuitry.
  • video or multimedia playback is typically implemented in software, such as via applications, applets or browser plug-ins. It should be understood, however, that the list of possible transmission, storage or processing technologies is merely illustrative and by no means exhaustive.
  • a key issue in the scenario shown in Figure 1 is that when the video repository VR is external to the client terminal CT, and the client terminal accesses the video repository over internet protocols, current technologies do not support discrete multichannel audio, wherein multichannel means more than the two channels of a stereo system.
  • a related problem is that the video repository VR may not offer audio in a language desired by the user of the client terminal.
  • the network environment shown in Figure 1 comprises an audio repository AR, from which audio content is retrieved to the client terminal CT.
  • a residual problem is how to synchronize the video content obtained from the video repository VR and audio content obtained from the audio repository AR.
  • the synchronization problems are most acute over poor remote connections, but they are not totally absent in cases where the video repository VR and/or the audio repository AR are stored locally in the client terminal.
  • the client terminal's processor may be multitasking, and another process requires processing power to the extent that playback may be interrupted for short intervals.
  • Figure 2 is a flow chart illustrating an operating principle according to an embodiment of the invention.
  • Figure 2 shows the processing steps carried out by the audio playback device. More particularly, Figure 2 shows the processing steps that relate to maintaining synchronization, and the steps of converting a digital audio stream to analog audio signals is presumed known to the skilled reader.
  • step 2-5 an audio player 3-200, acting as an HTTP server, receives vid- eo-related time-code information, called first time-code information in the following, over an application programming interface (API, see item 3-150 in Figure 3).
  • the audio player obtains audio-related time-code information, called second time-code information.
  • step 2-15 the audio player determines a time difference between the second and first time-code information of concurrently outputted audio and video streams.
  • steps 2-20 comprises calculating a time-filtered version, such as one or more moving average functions, from the time difference determined in the preceding step.
  • step 2-25 the time-filtered version(s), such as the moving average function (s) is/are analyzed.
  • step 2-25 If the analysis in step 2-25 shows that audio and video are synchronized with each other, the process proceeds to step 2-30, wherein the incoming audio samples are processed (sampled) normally. This is the normal processing of audio samples that is presumed known to the skilled reader. In other words, step 2- 30 performs what a conventional audio player does.
  • step 2-25 If the analysis in step 2-25 shows that audio leads (is ahead of) video, the process proceeds to step 2-35, wherein the incoming audio samples are over- sampled. Oversampling of incoming audio samples has the effect that processing of a given set of audio samples lasts longer than by normal processing, and the audio leads video by a progressively smaller amount of time. Over time, the time difference determined in step 2-15 changes sign and the moving average (s) calcu- lated in step 2-20 approach zero. If the audio signal leads video by a significant margin, the process may repeat entire frames. Each repeated frame doubles the amount of time that frame is outputted, and reduces the time that audio is ahead of video by the duration of the frame.
  • step 2-25 shows that audio trails (is behind) video
  • the process proceeds to step 2-40, wherein the incoming audio samples are undersampled. Undersampling of incoming audio samples has the effect that processing of a given set of audio samples lasts less time than by normal processing, and the audio trails video by a progressively smaller amount of time. Over time, the time difference determined in step 2-15 changes sign and the moving average(s) calculated in step 2-20 approach zero. If the audio signal trails video by a significant margin, the process may drop entire frames. Each dropped frame advances audio (reduces lag) in relation to video by the duration of the frame.
  • FIG 3 is a block diagram illustrating an exemplary hardware implementation of a data processing device, such as the client terminal CT shown in Figure 1.
  • Reference numeral 3-100 denotes a video player, which may be imple- mented as an external application (a HTTP client) to the audio player acting as an HTTP Server 3-200.
  • Outputs of a typical video player 3-100 include a video output 3-130 and an audio output 3-132, which the video player 3-100 synchronizes with each other by an internal audio-to-video synchronization mechanism.
  • the video output 3-130 of the video player 3- 100 is utilized normally, that is, typically outputted to an output device.
  • the video player 3-100 also generates an audio output 3-132, it may be replaced by a different audio output, denoted by reference number 3-230, which is based on a different audio stream, denoted by reference number 3-202.
  • Processing of the incoming audio stream 3-202 in the audio player 3-200 involves the following blocks. Dashed outline around some of the blocks indicates that the block is involved in audio-video synchronization and not merely to con- ventional audio processing.
  • a source buffer or cache 3-210 permits the audio player to store incoming audio packets, by virtue of which the audio player can survive brief communication delays without audible gaps in outputted audio.
  • An AV demultiplexer extracts a selected audio stream from an incoming AV bitstream, which may contain multiple bitstreams, such as audio streams in several languages.
  • Reference number 3- 214 denotes a frame dropper, which is used in situations wherein audio significantly trails video. By dropping audio for some frames, the outputted audio can be made to quickly catch up with the video. Operation of the frame dropper 3-214 is controlled by the synchronization controller 3-240.
  • the audio frames are conveyed to an audio decoder 3-216 which in this example outputs pulse-code modulated (PCM) frames to a multichannel mixer 3-218 for downmixing or upmixing of audio.
  • Output from the multichannel mixer 3-218 is conveyed to a frame repeater 3-220, whose function is opposite to that of the frame dropper 3-214.
  • the frame repeater 3-220 repeats some of the frames, thus spending more than a normal amount of time to process a series of audio frames. As a result the outputted audio can be made to slow down to match the time code of the currently played video.
  • Reference number 3-222 denotes a sample rate converter, which the synchronization controller 3-240 acti- vates in cases where the outputted audio leads or trails video by an amount which is not large enough to activate the frame dropper 3-214 or the frame repeater 3- 220.
  • the sample rate converter has the effect that, instead of dropping or repeating whole frames, individual frames are played for a longer or shorter time, depending on whether the outputted audio leads or trails the video.
  • the PCM frames processed by the audio player 3-200 are conveyed to an audio playback device generally denoted by reference number 3-224 and outputted as audio output 3-230.
  • the audio player / HTTP Server 3-200 offers an Application Programming Interface ["API"] 3-150 to the video player / HTTP client 3-100.
  • RESTful signifies that the software architecture conforms to REST constraints.
  • REST refers to "REpresentational State Transfer", which is a style of software architecture for distributed systems such as the World Wide Web.
  • REST is by no means the only design model, it has emerged as a predominant model for Web service design.
  • the term "representational state transfer” was introduced in the doctoral dissertation of Roy Fielding, who is one of the principal authors of the Hypertext Transfer Protocol (HTTP) specification versions 1.0 and 1.1.
  • the video player provides the audio player 3-200 with audio- related time code 3-160, which may be the audio output 3-132 itself or some related or derivative timing signal from the audio output 3-132.
  • the synchronization controller 3-240 obtains the audio-related time code 3-160 via the API 3-150, and calculates a time-delay estimation (TDE).
  • TDE time-delay estimation
  • An exemplary technique for the TDE estimation is based on Generalized Cross-Correlation (GCC). This technique is presented in reference document #1 (identified at the end of this description).
  • some implementations of the invention may deliver the audio output signal 3-132 to the synchronization controller 3-240.
  • the synchronization controller 3-240 is then able to calculate a time-delay estimate (TDE) between the audio output 3-132 of the video player 3-100 and the audio output 3-230 of the audio player 3-200, or some internally processed signal, such as the output of the audio decoder 3-216.
  • TDE time-delay estimate
  • Web Audio API which describes a high-level JavaScript API for processing and synthesizing audio in web applications.
  • IETF Internet Engineering Task Force
  • RFC 6455 specifies WebSocket protocol.
  • the WebSocket Protocol enables two-way communication between a client and a server.
  • the WebSocket Protocol is an independent TCP-based protocol, but its handshake is interpreted by HTTP servers as an Upgrade request.
  • Some implementations of the video player might use these two protocols to route the audio output 3-132 by using WebSocket to the synchronization controller 3-240.
  • both audio signals may be downmixed to a single channel (mono).
  • a stereo signal can be downmixed to mono by summing the left and right channel, and attenuating the signal by -6 dB, or by any of a number of equivalent processes.
  • Some implementations of the invention are based on the assumption that if a multichannel audio content is available, that multichannel audio content was used to produce the downmixed stereo.
  • the downmixing scheme (downmixing matrix) used to produce the stereo may also be known and this knowledge may be used to correctly downmix the multichannel audio to stereo.
  • the downmixing scheme is not known.
  • the downmixing scheme may be assumed to be one of Dol- by® Surround, Dolby® Pro Logic or Dolby® Pro Logic II scheme, which are the ones most widely used in this field, all being registered trademarks of Dolby Laboratories, Inc.
  • the use of these schemes may be detected using a decoder of the downmixing scheme.
  • the number of channels in multichannel content may be used to determine the downmixing scheme for multichannel audio.
  • phase transformation may be used as the frequency weighting function of GCC.
  • GCC-PHAT phase transformation
  • the GCC-PHAT is defined as:
  • X t (f) and X j (f) are the Fourier transforms of the two signals, while []* denotes the complex conjugate.
  • the delay for these signals can be estimated as:
  • R PHAT (d) is is the inverse Fourier transform of G PHAT (f).
  • a cross correlation threshold is the minimum correlation value at which the correlation is expected to return feasible results.
  • Efficient computation of GCC-PHAT may be ac- complished by using zero padded Fast Fourier Transform (FFT), scaling and inverse FFT.
  • FFT Fast Fourier Transform
  • An FFT function is known to have a complexity of 0(N log N).
  • As the aim of this technique synchronize playback, it is to run in real time. If delay estimation of longer delays is required, a longer window is required, which increases the required amount of processing.
  • the maximum delay estimation using GCC-PHAT is limited by the processing capabilities of the device. But as this synchronization method is secondary to time-code based synchronization, the maximum window size can be adapted to the processing capabilities, while switching between synchronization methods can be done accordingly.
  • Reference numeral 3-250 denotes a system hardware clock of the data processing device CT. Although the system hardware clock may be used for other purposes, it is an unsuitable clock for the purpose of synchronizing audio and video. Instead, a synchronization controller denoted by reference numeral 3-240 performs the audio-to-video synchronization.
  • the presently described embodiment utilizes a clock synchronization algorithm based on an internal clock model, where a goal is to minimize the maximum difference between two clocks.
  • Clock synchronization is asymmetric, which is also known as a master-slave synchronization scheme.
  • the video player is the master while the audio player is the slave.
  • the presently described embodiment requires minimal changes to the video player 3-100, compared with a typical prior art video player.
  • the clock synchronization process uses a technique commonly known as a slave-controlled scheme, where the master (the video player 3-100) provides a reference time to which the slave (the audio player 3-200) synchronizes.
  • the embodiment of Figure 3 imposes few if any restrictions regarding the source from which the video player receives the video content or the manner in which the video content is retrieved.
  • the audio player 3-200 and the video player 3-100 should use the same time reference.
  • the video player 3-100 receives a packetized stream of multiplexed audio and video in an MPEG-2 or MPEG-4 container. These container formats may define several different time stamps to control playback rate. Normally the video player 3-100 merely needs to read the time stamps and wait until the right time to output each video frame on the display (inherent feature in a video player, not shown separately).
  • abnormal situations may occur that cause one or more video frames being displayed at an incorrect time or for an incorrect duration, and this results in slower or faster playback of the video.
  • Ex- amples of such abnormal situations include system overload, transmission delays, reception errors, or the like. Precise reasons and natures of failed synchronization are irrelevant for the purposes of the present invention, however.
  • a video player may compensate for such abnormal situations by adjusting the presentation time of each frame.
  • a residual problem in this kind of compensation technique is crea- tion of deviation (jitter) in the master clock.
  • the embodiment of Figure 3 begins a clock synchronization process in such a manner that the video player 3-100 sends its current presentation timestamp (a timestamp of a currently displayed video frame) to the audio player 3-200 using the HTTP Control API 3-150.
  • video-related time stamp information is called first time code information.
  • the video player 3-100 is expected to continue sending the first time code information as long as the video is being played. It is not necessary to send the first time code information for each video frame being played, but it is beneficial to define an upper limit for the interval between transmissions of the first time code information. For instance, the upper limit may range from one to a few sec- onds, such as two seconds. In an illustrative but non-restrictive example, the video player 3-100 may use watchdog timers to send the first time code information.
  • the clock synchronization controller block 3-240 receives the video-related first time-code information and employs a clock synchronization algorithm, which will be described in the following.
  • the clock synchronization algorithm employed by the clock synchronization controller 3-240 compares the first time code information V(i) and the second time code information (the audio-related timestamp) A(i) with corresponding previous timestamps V(i-l) and A(i-l) to calculate an estimate of the clock deviation:
  • the video player 3-100 is known or expected to have jitter on its timestamps, and since network latencies and operating system schedulers add to this jitter, the estimate of the clock deviation D(i) is smoothed by using moving average filter (s).
  • moving average filter s
  • An exponential moving average also known as an exponentially weighted moving average (EWMA) is a type of infinite impulse response filter that applies exponentially decreasing weighting factors. Although the weighting for each older datum point decreases exponentially, it never reaches true zero in a mathematically exact description.
  • the EMA for a series D may be calculated recursively
  • the coefficient alpha represents the degree of weight decrease, a constant smoothing factor between 0 and 1.
  • a more ambitious implementation of the clock synchronization algorithm employs several different weighting factors to estimate clock deviation in differ- ent time periods.
  • alpha and beta are positive integers.
  • alpha may have a value of 1, while beta has val- ues 3,15, 31 for three different time periods.
  • the resulting filtered value gives an estimation of clock skew. Based on this estimated clock skew, the synchronization controller 3-240 decides to drop or repeat audio frames in order to compensate for the clock skew. If the clock skew is too large, the synchronization controller 3-240 instructs the A/V demulti- plexer 3-212 to seek to a new position in the audio content. When the new position has been achieved, and the clock skew has been reduced to a few hundred milliseconds at most, the sample rate converter 3-222 is used to compensate for the clock skew. As long as the sample rate variations is low enough, the change in pitch produced by the sample rate conversion will not be audible.
  • JND just noticeable difference
  • the synchronization controller 3-240 may stop correcting for clock skew when the clock skew estimation changes its sign. In a representative but non-restrictive implementation, as long as the absolute value of the clock skew remains below 20 milliseconds, the synchronization controller 3-240 lets the audio play normally without making any time-related corrections.
  • Figure 4 is another block diagram showing major hardware blocks in a data processing device.
  • Figure 4 schematically shows an exemplary block diagram for a data processing device 4-100, such as the client terminal CT, which includes the HTTP server acting as the audio player.
  • the data processing device comprises one or more central processing units CP1 ... CPn, generally denoted by reference numeral 4-110.
  • Embodiments comprising multiple processing units 4-110 are preferably provided with a load balancing unit 4-115 that balances processing load among the multiple processing units 4-110.
  • the multiple processing units 4- 110 may be implemented as separate processor components or as physical pro- cessor cores or virtual processors within a single component case.
  • the data processing device 4-100 further comprises a network interface 4-120 for communi- eating with various data networks, which are generally denoted by reference sign DN.
  • the data networks DN may include local-area networks, such as an Ethernet network, and/or wide-area networks, such as the internet.
  • the data processing device may connect to online video and/or audio repositories via data networks DN.
  • the data processing device 4-100 of the present embodiment also comprises a user interface 4-125.
  • the user interface comprises a display.
  • the nature of the user interface depends on which kind of computer is used to implement the data processing device 4-100.
  • the user inter- face 4-125 may comprise a traditional keyboard, mouse and/or touchpad, and/or a touch-sensitive display as used in tablet computers.
  • the data processing device 4-100 also comprises memory 4-150 for storing program instructions, operating parameters and variables.
  • Reference numeral 4-160 denotes a program suite for the data processing device 4-100.
  • the data processing device 4-100 also comprises circuitry for various clocks, interrupts and the like, and these are generally depicted by reference numeral 4-130.
  • the data processing device 4-100 further comprises a disk interface to the disk system 4-190.
  • the various elements 4-110 through 4-150 intercommunicate via a bus 4-105, which carries address signals, data signals and control signals, as is well known to those skilled in the art.
  • the inventive method may be implemented in the data processing device as follows.
  • the program suite 4-160 comprises program code instructions for instructing the set of processors 4-110 to execute the functions of the inventive method, wherein the functions include performing common operating system functions, normal data processor functions, including data entry, presentation and communication.
  • the functions include video player functionality, audio player functionality and clock synchronization.
  • the functions of the inventive method include the acts defined in claim 1.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

L'invention concerne la synchronisation d'audio et de vidéo, comprenant le traitement de contenu vidéo par un client HTTP dans un dispositif de traitement de données. Le contenu vidéo comprend des trames de données vidéo avec un premier code temporel. Une application de serveur HTTP dans le dispositif (CT) offre une interface de programmation d'application [« API »] au client HTTP et reçoit (2-10) du contenu audio, qui comprend des échantillons de données audio associés à un second code temporel. Le serveur HTTP produit des trames audio à modulation par impulsions et codage [« MIC »] à partir des échantillons de données audio ; reçoit (2-5) le premier code temporel en provenance du client HTTP sur l'API ; détermine d'une manière répétée (2-15) une différence de temps entre les second et premier codes temporels ; et calcule (2-20) une moyenne mobile de la différence de temps déterminée d'une manière répétée. Si la fonction moyenne mobile indique que les trames de données audio MIC sont en avance ou en retard sur les trames de données vidéo délivrées, les trames de données audio MIC sont sur-échantillonnées (2-30) ou sous-échantillonnées (2-40) respectivement, et autrement elles sont échantillonnées normalement (2-45).
PCT/FI2014/050138 2013-02-21 2014-02-21 Synchronisation de contenu audio et vidéo Ceased WO2014128360A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
FI20135153 2013-02-21
FI20135153 2013-02-21

Publications (1)

Publication Number Publication Date
WO2014128360A1 true WO2014128360A1 (fr) 2014-08-28

Family

ID=51390555

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2014/050138 Ceased WO2014128360A1 (fr) 2013-02-21 2014-02-21 Synchronisation de contenu audio et vidéo

Country Status (1)

Country Link
WO (1) WO2014128360A1 (fr)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109040717A (zh) * 2018-10-12 2018-12-18 厦门美亚中敏科技有限公司 一种指挥调度信息展示方法及系统
CN111131917A (zh) * 2019-12-26 2020-05-08 国微集团(深圳)有限公司 音频频谱实时同步方法、播放装置
CN114025229A (zh) * 2021-11-09 2022-02-08 上海爱奇艺新媒体科技有限公司 处理音视频文件的方法、装置、计算设备及存储介质
WO2022049210A1 (fr) * 2020-09-02 2022-03-10 Effing Simon Procédé de synchronisation de la lecture d'un fichier audio avec un fichier vidéo associé
CN116801021A (zh) * 2023-06-08 2023-09-22 北京花房科技有限公司 分布式的流媒体播放系统、方法、设备及存储介质
CN118338093A (zh) * 2024-06-14 2024-07-12 杭州阿启视科技有限公司 一种基于web前端播放H.265视频流的软解方法
WO2025006352A1 (fr) * 2023-06-29 2025-01-02 Google Llc Systèmes et procédés de synchronisation de flux multimédias codés indépendamment

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6122668A (en) * 1995-11-02 2000-09-19 Starlight Networks Synchronization of audio and video signals in a live multicast in a LAN
US20080219637A1 (en) * 2007-03-09 2008-09-11 Sandrew Barry B Apparatus and method for synchronizing a secondary audio track to the audio track of a video source
US20120039582A1 (en) * 2009-04-20 2012-02-16 Koninklijke Philips Electronics N.V. Verification and synchronization of files obtained separately from a video content
WO2012049223A2 (fr) * 2010-10-12 2012-04-19 Compass Interactive Limited Son alternatif
US20120307149A1 (en) * 2007-02-28 2012-12-06 At&T Intellectual Property I, L.P. Methods, Systems, and Products for Alternate Audio Sources

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6122668A (en) * 1995-11-02 2000-09-19 Starlight Networks Synchronization of audio and video signals in a live multicast in a LAN
US20120307149A1 (en) * 2007-02-28 2012-12-06 At&T Intellectual Property I, L.P. Methods, Systems, and Products for Alternate Audio Sources
US20080219637A1 (en) * 2007-03-09 2008-09-11 Sandrew Barry B Apparatus and method for synchronizing a secondary audio track to the audio track of a video source
US20120039582A1 (en) * 2009-04-20 2012-02-16 Koninklijke Philips Electronics N.V. Verification and synchronization of files obtained separately from a video content
WO2012049223A2 (fr) * 2010-10-12 2012-04-19 Compass Interactive Limited Son alternatif

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109040717A (zh) * 2018-10-12 2018-12-18 厦门美亚中敏科技有限公司 一种指挥调度信息展示方法及系统
CN111131917A (zh) * 2019-12-26 2020-05-08 国微集团(深圳)有限公司 音频频谱实时同步方法、播放装置
CN111131917B (zh) * 2019-12-26 2021-12-28 国微集团(深圳)有限公司 音频频谱实时同步方法、播放装置
WO2022049210A1 (fr) * 2020-09-02 2022-03-10 Effing Simon Procédé de synchronisation de la lecture d'un fichier audio avec un fichier vidéo associé
US12342056B2 (en) 2020-09-02 2025-06-24 Nubart Gmbh Method for synchronizing the playback of an audio file with an associated video file
CN114025229A (zh) * 2021-11-09 2022-02-08 上海爱奇艺新媒体科技有限公司 处理音视频文件的方法、装置、计算设备及存储介质
CN114025229B (zh) * 2021-11-09 2025-01-21 上海爱奇艺新媒体科技有限公司 处理音视频文件的方法、装置、计算设备及存储介质
CN116801021A (zh) * 2023-06-08 2023-09-22 北京花房科技有限公司 分布式的流媒体播放系统、方法、设备及存储介质
WO2025006352A1 (fr) * 2023-06-29 2025-01-02 Google Llc Systèmes et procédés de synchronisation de flux multimédias codés indépendamment
US12328471B2 (en) 2023-06-29 2025-06-10 Google Llc Systems and methods for synchronization of independently encoded media streams
CN118338093A (zh) * 2024-06-14 2024-07-12 杭州阿启视科技有限公司 一种基于web前端播放H.265视频流的软解方法

Similar Documents

Publication Publication Date Title
KR102393798B1 (ko) 오디오 신호 처리 방법 및 장치
JP6640359B2 (ja) ワイヤレスオーディオ同期
JP5149012B2 (ja) ネットワーク上のマルチチャネルスピーカの同期
US9684485B2 (en) Fast-resume audio playback
EP2752023B1 (fr) Procédé pour mettre en correspondance des estampilles temporelles d'entrée et de sortie dans un encodeur vidéo et un dispositif d'insertion de publicité
JP5273858B2 (ja) データストリームおよびマルチチャネル表現を生成するための装置および方法
US10856018B2 (en) Clock synchronization techniques including modification of sample rate conversion
US7647229B2 (en) Time scaling of multi-channel audio signals
JP2019193268A (ja) メディア信号を復号化する復号器及び、一次メディアデータについてのメタデータ又は制御データを含む二次メディアデータを符号化する符号器
JP6290915B2 (ja) 共通イベントベースのマルチデバイスメディア再生
US9843489B2 (en) System and method for synchronous media rendering over wireless networks with wireless performance monitoring
CN106688251A (zh) 音频处理系统和方法
KR20060125678A (ko) 버퍼 관리 시스템, 디지털 오디오 수신기, 헤드폰들,확성기, 버퍼 관리 방법
US10805664B2 (en) Wireless audio synchronization
US9967437B1 (en) Dynamic audio synchronization
CN103460128A (zh) 借助智能电话和音频水印的多种语言同步电影配音
US9118678B2 (en) Indirect clock measuring and media adjustment
US11323780B2 (en) Systems and methods for determining delay of a plurality of media streams
JP7365212B2 (ja) 動画再生装置、動画再生システム、および動画再生方法
CN108632557B (zh) 一种音视频同步的方法及终端
KR20070008069A (ko) 음성/영상신호의 동기화 장치 및 방법
JP6596363B2 (ja) 時刻マッピング情報生成装置、同期再生システム、時刻マッピング情報生成方法及び時刻マッピング情報生成プログラム
GB2596107A (en) Managing network jitter for multiple audio streams
EP3477887A1 (fr) Dispositif de réglage de synchronisation, système de distribution, procédé de réglage de synchronisation et programme
KR101810883B1 (ko) 라이브 스트리밍 시스템 및 그의 스트리밍 클라이언트

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14753459

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14753459

Country of ref document: EP

Kind code of ref document: A1