Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H1/00—Details of electrophonic musical instruments
  • G10H1/36—Accompaniment arrangements
  • G10H1/361—Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
  • G10H1/368—Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems displaying animated or moving pictures synchronized with the music or audio part
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H1/00—Details of electrophonic musical instruments
  • G10H1/36—Accompaniment arrangements
  • G10H1/361—Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H1/00—Details of electrophonic musical instruments
  • G10H1/36—Accompaniment arrangements
  • G10H1/361—Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
  • G10H1/365—Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems the accompaniment information being stored on a host computer and transmitted to a reproducing terminal by means of a network, e.g. public telephone lines
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/02—Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
  • G11B27/031—Electronic editing of digitised analogue information signals, e.g. audio or video signals
  • G11B27/036—Insert-editing
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/64—Circuits for processing colour signals
  • H04N9/74—Circuits for processing colour signals for obtaining special effects
  • H04N9/75—Chroma key
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
  • G10H2220/005—Non-interactive screen display of musical or status data
  • G10H2220/011—Lyrics displays, e.g. for karaoke applications
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
  • G10H2220/155—User input interfaces for electrophonic musical instruments
  • G10H2220/441—Image sensing, i.e. capturing images or optical patterns for musical purposes or musical control purposes
  • G10H2220/455—Camera input, e.g. analyzing pictures from a video camera and using the analysis results as control data
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/222—Studio circuitry; Studio devices; Studio equipment
  • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
  • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay

Definitions

• the present disclosure relates to an interactive audio/video system and method.
• the present disclosure relates to an interactive karaoke audio/video system and method.
• Karaoke is a popular pastime and there is a need for a system and method for making a realistic audio/video performance that incorporates images and voices of the participants, in real time, within an audio/video clip.
• the present disclosure provides an interactive audio/video system for making an interactive audio/video clip of at least one participant, comprising:
• a screen of a color compatible with a Chroma keying process at least one neck and torso bib of a color compatible with a
• an audio/video database containing at least one audio/video clip and an associated file containing text associated with the audio/video clip
• a audio/video server in communication with the at least one microphone, the at least one camera, the display and the audio/video database, the audio/video server being configured to: access the at least one audio/video clip and the file associated with the at least one audio/video clip from the audio/video database;
• an interactive audio/video system for making an interactive audio/video clip as above wherein the file associated with the audio/video clip is a video file that comprises:
• the audio/video server is further configured to perform the step of superimposing the isolated head of the at least one participant on the body of the associated character in synchronization with the head appearance timing reference information of the associated character.
• an interactive audio/video system for making an interactive audio/video clip as above wherein the file associated with the audio/video clip further comprises:
• At least one additional audio track having therein encoded information for the associated character selected from the group consisting of voice enabling timing information, angle of view timing reference information and positioning timing reference information;
• audio/video server is further configured to perform the steps of:
• the present disclosure also provides a corresponding method of making an interactive audio/video clip of at least one participant.
• FIG. 1 is a schematic representation of an interactive audio/video system in accordance with an illustrative embodiment of the present disclosure
• FIG. 2 is a schematic representation of an exemplary architecture of the audio/video server of FIG. 1;
• FIG. 3 is a schematic representation of interactive audio/video systems in communication with a remote audio/video clip distribution server
• FIG. 4 is a flow diagram of an interactive audio/video process in accordance with an illustrative embodiment of the present disclosure
• FIG. 5 is a flow diagram of an illustrative example of the audio mixing process used by the interactive audio/video process
• FIG. 6 is a flow diagram of an illustrative example of the video mixing process used by the interactive audio/video process.
• FIG. 7 is a flow diagram of an illustrative example of the recording process used by the interactive audio/video process.
• the non-limitative illustrative embodiments of the present disclosure provide an interactive audio/video system and method for integrating, in real time, the singing and/or acting performance of participants into a selected audio/video clip.
• the interactive audio/video system 10 generally consists of one or more selection station 12, an audio/video server 14 with an associated audio/video database 16, a green screen 18, one or more green neck and torso bib 20, one or more microphone 22, one or more video camera 24 and a display screen 26.
• the one or more selection station 12 is used by participants 1 to select one or more available audio/video clips from the audio/video database 16. Participants 1 may also enter personal information such as name(s), e-mail address(es), etc.
• the selection station 12 can be a dedicated station with an input/output interface and a communication system to communicate, wirelessly or otherwise, with the audio/video server 14 or a personal computer, lap top computer, tablet device, personal digital assistant, intelligent phone or any other such device.
• the selected audio/video clips may be ghosted in the lists of available audio/video clips for a pre-determined time period in order to avoid a single audio/video clip from being selected repeatedly by multiple participants 1.
• the interactive audio/video system 10 may, optionally, further comprise a remote audio/video dip distribution server 34, with an associated remote audio/video clip database 36, accessible from audio/video servers 14 via a wide area network (WAN) 30 such as, for example, Ethernet (broadband, high-speed), wireless WiFi, cable Internet, satellite connection, cellular or satellite network, etc. Audio/video clips with their associated information may be provided by the remote audio/video clip distribution server 34 for remote download to any connected audio/video servers 14 via, for example, an online store application or as part of a subscription agreement.
• WAN wide area network
• the audio/video server 14 includes a processor 40 with an associated memory 50 having stored therein processor executable instructions for configuring the processor 40 to perform various processes, namely an interactive audio/video process 51 , an audio mixing process 52, a video mixing process 53 and a recording process 54, which processes will be further described below.
• the audio/video server 14 further includes an input/output (I/O) interface 42 for communication with the various components of the interactive audio/video system 10 and, optionally, a remote audio/video clip distribution server 34.
• I/O input/output
• the audio/video server 14 receives participants' 1 selections from the one or more selection station 12 and automatically sorts the selected audio/video clips by the time they were selected. The next audio/video clip in the list along with its associated information is accessed from the audio/video database 16 and the participant 1 having selected the audio/video clip is invited, along with any other associated participants 1 , to take position in front of the green screen 18.
• the participants 1 are each provided with a green neck and torso bib 20 and a microphone 22, for example a wireless microphone, in communication with the audio/video server 14.
• the one or more audio/video camera 24 is aimed at the green screen 18 and is in communication with the audio/video server 14 in order to capture images of the participants 1.
• the heads of each of the participants 1 are isolated from the audio/video feed, for example using a Chroma keying process, and superimposed on the body of an associated character in the selected audio/video clip. The size of the heads may be adjusted in order to properly fit with the body of the associated character. It is to be understood that the heads of the participants 1 may be switched between the various characters if desired.
• the selected audio/video clip with the superimposed heads is displayed on the display screen 26 so that the participants 1 may view their live performance as well as the lyrics or dialogue associated with the audio/video clip.
• the participants 1 sing along, or recite dialogue, to the audio/video clip's lyrics or dialogue, reading from the display screen 26 while viewing their performance in real time, their voices and image being recorded.
• additional display screens may be added in order to display the participants' 1 performance to an audience.
• the audio/video clips in the illustrative embodiment are created by superimposing two video layers: a first layer consisting of a background and a second layer consisting in one or more characters, and one or more audio tracks containing a musical score and/or other background sounds.
• a first layer consisting of a background
• a second layer consisting in one or more characters
• one or more audio tracks containing a musical score and/or other background sounds With each audio/video clip there is also stored in the audio/video database 16 an associated file containing the written lyrics or dialogue, or other text, for the audio/video clip.
• the lyrics/dialogue file is a video file comprising a video layer with the lyrics/diaiogue and one of more audio tracks used to encode head appearance timing reference information for each character. It is to be understood that in an alternative embodiment the head appearance timing reference information may be omitted.
• further audio tracks may be used to encode voice enabling timing reference information and/or angle of view timing reference information for each character.
• the voice and head appearance enabling and the angle of view timing reference information may be encoded, for example, by having a signal (e.g. high frequency noise) on an associated audio track above a certain threshold for enablement/selection.
• the level of the signal may be indicative of the volume of an associated microphone 22 and the transparency applied to an associated video camera 24.
• different frequencies may be used for associated characters, microphones 22 and video cameras 24.
• the lyrics/dialogue file may also include audio tracks used to encode positioning timing reference information for some or all of the characters appearing in the audio/video clip in order to automatically displace the heads of corresponding participants 1 to follow movements of the characters.
• This encoded information may be in the form of, for example, KinectTM data.
• the audio/video clips may be computer generated. In another alternative embodiment the audio/video clips may be preprocessed music videos, movie scenes, etc.
• one or more logo may be added on a third audio/video layer or may be provided as an associated image file.
• the audio/video clips and lyrics/dialogue files may be encoded, for example, in the MP4 format.
• the audio/video clips and lyrics/dialogue video files stored in the audio/video database 16 may be encrypted using, for example, a 128bit key based on a serial number of the associated audio/video server 14 so as to be used only by that specific audio/video server 14.
• the key may be such as to allow use of the audio/video clips and lyrics/dialogue files by a specific set of audio/video servers 14.
• audio/video clips and lyrics/dialogue files stored in the remote audio/video clip database 36 may be encrypted similarly by the associated remote audio/video clip distribution server 34 upon a request for download by an audio/video server 14.
• the interactive audio/video system 10 uses the timing reference information provided for each audio/video clip, synchronizes, for each participant 1, the appearance of the participant's 1 head and, optionally, the enablement of the recording of the voice and the selection of the angle of view of each of the participants 1 with the appearance and singing or talking of an associated character in the audio/video clip. Voice effects for each participant's 1 voice may also be added in real time. It is to be understood that each participant 1 may be associated with one or more characters in the audio/video clip and that the specific associations may be changed during the audio/video performance.
• the voice enabling timing reference information is used to enable the audio signal from the microphone 22 associated with each participant 1 and the angle of view timing reference information is used to select from which video camera 24 the image of each of the participant's 1 head should be taken from.
• the participants' 1 performance can be recorded so as to be saved on a DVD, USB key or other such memory support medium and provided to the participants 1 at the conclusion of their performance.
• the participants' 1 performance can also be saved on a flash drive, hard drive, computer memory, etc.
• the recorded performance can then be provided to the participants 1 via e-mail, file transfer protocole (FTP) or any other such data transfer or data uploading/downloading services.
• FTP file transfer protocole
• the participants' 1 performance may further be uploaded to social networking or sharing sites such as facebookTM, youtubeTM, etc.
• social networking or sharing sites such as facebookTM, youtubeTM, etc.
• one or more logo for example of one or more sponsor may be added to the recorded performance.
• the interactive audio/video system 10 may be provided with a secure payment system, which may be implemented on the one more selection station 12, the audio/video server 14, or be a stand-alone system. In an alternative embodiment, payment may also be provided by phone or Internet
• FIG. 4 there is shown a flow diagram of an illustrative example of the interactive audio/video process 100 executed by the audio/video server 14. Steps of the process 100 are indicated by blocks 102 to 116.
• the process 100 starts at block 102 where the selected audio/video clip and associated lyrics/dialogue file containing the timing reference information are accessed from the audio/video database 6.
• the audio/video clip with its lyrics or dialogue, are displayed on the display screen 26 and, at block 106, the live performance of the participants 1 is provided to the audio/video server 14 via the one or more microphone 22 and camera 24.
• the recorded audio and video, respectively, portions of the live performance are mixed in real time with the audio and video of the audio/video clip.
• the audio and video mixing processes will be further detailed below.
• the mixed audio and video streams are combined and recorded to produce the interactive audio/video clip.
• the recording process will be further detailed below.
• the process 100 verifies if the audio/video clip is at its end, if so it proceeds to 116 where the completed interactive audio/video is provided to the participant(s), if not, it proceeds back to block 104.
• FIG. 5 there is shown a flow diagram of an illustrative example of the audio mixing process 200 used by the interactive audio/video process 100 of FIG. 4. Steps of the process 200 are indicated by blocks 202 to 220.
• the process 200 starts at blocks 202 and 204 where the audio signal from the voice or each participant 1 is inputted from its associated microphone 22. It is to be understood that the number of inputs varies according to the number of microphones 22.
• the process 200 verifies, for each participant 1, if the voice timing reference information indicates that the character associated with the participant 1 is singing or talking in the audio/video clip. If so, the process 200 enables the associated microphone 22 input and proceeds to block 210, if not, it simply continues monitoring the voice timing reference information untit it indicates that the specific microphone 22 input should be enabled.
• steps 206 and 208 will consequently not be present as well.
• audio effects may be added to the voices of the participants 1 , for example reverb, echo, etc. These effects may be audio/video clip dependent or operator selectable. It is to be understood that this step may be optional, in which case steps 210 and 212 may be omitted and the mixing effectuated at block 218. It is also to be understood that in an alternative embodiment the audio effects may be added independently to the voice of each participant 1 , in which case block 212 will be replaced by corresponding blocks after each individual microphone input 202, 204.
• the audio/video clip is provided to the audio mixing process 200 following which, at block 216, the audio is extracted from the audio/video clip.
• the voices of the participants 1 are mixed with the audio of the audio/video clip extracted at block 216.
• Steps of the audio mixing process 200 may be performed using, for example, AudiolabTM components.
• FIG. 6 there is shown a flow diagram of an illustrative example of the video mixing process 300 used by the interactive audio/video process 100 of FIG. 4. Steps of the process 300 are indicated by blocks 302 to 332.
• the process 300 starts at blocks 302 and 304 where the video image of the participants 1 is inputted from the video cameras 24. It is to be understood that the number of inputs varies according to the number of video cameras 24.
• green screen keying is applied to the video images provided by the video camera 24 inputs of blocks 302 and 304, following which, at blocks 310 and 312, the heads of each of the participants 1 are isolated from the recorded audio/video performance. If multiple video cameras 24 are used, the angle of view timing reference information is used to select from which video camera 24 the image of each of the participant's 1 head should be taken from.
• the process 300 verifies, for each participant 1 , if the head appearance timing reference information indicates that the character associated with the participant 1 is appearing in the audio/video clip. If so, the process 300 enables the video image of the head of the participant 1 and proceeds to block 320, if not, it simply continues monitoring the head appearance timing reference information until it indicates that the head of the specific participant 1 should appear (i.e. its associated character is present in the audio/video clip).
• the audio/video clip is provided to the video mixing process 300.
• the images of the heads of the participants 1 whose appearance have been enabled at blocks 314 and 316 are mixed with the audio/video clip provided at block 318 so as to be superimposed on the body of the associated character.
• the mixed video stream of the participants' 1 performance is provided and, optionally, at block 324, a logo is added to the video stream.
• the lyrics/dialogue file associated with the audio/video clip is provided, at block 326 to the video mixing process 300, following which, at block 328, green screen keying is applied to the video file in order to isolate the lyrics/dialogue.
• Step 330 the isolated lyrics/dialogue from block 328 are mixed together with the video stream from block 322 in order to be displayed, at block 332, on the display screen 26 so that the participants 1 may view their live performance as well as the lyrics or dialogue associated with the audio/video clip but not recorded with the live performance.
• Steps of the video mixing process 300 may be performed using, for example, MediaLooksTM components.
• FIG. 7 there is shown a flow diagram of an illustrative example of the recording process 400 used by the interactive audio/video process 100 of FIG. 4. Steps of the process 400 are indicated by blocks 402 to 420.
• the process 400 starts at blocks 402 and 404 where the audio stream from process 200 of FIG. 5 and the video stream from process 300 of FIG. 6 are provided to the recording process 400.
• the audio and video streams from blocks 402 and 404 are multiplexed together, for example using an advanced streaming format (ASF) multiplexor, to produce, at block 408, a high definition (HD) file of the interactive audio/video clip containing the performance of the participants 1, for example a windows media video ⁇ WMV) file.
• ASF advanced streaming format
• HD high definition
• the HD file of the interactive audio/video clip is then saved, at block 410, for example to a drive, flash memory, etc., and, at block 412, uploaded to a hosting service in order to me remotely accessible via, for example FTP or social networking or sharing sites such as facebookTM, youtubeTM, etc..
• the HD file of the interactive audio/video clip is converted in order to produce a low definition (LD) file of the interactive audio/video clip which, at block 416, can be transmitted to the participants 1 via, for example, e-mail or short message service (SMS) using information initially provided by the participants 1 at one of the selection stations 12.
• LD low definition
• SMS short message service
• the LD file of the interactive audio/video clip is provided to a DVD author in order to produce, at block 420, a DVD of the interactive audio/video clip containing the performance of the participants 1.
• HD and/or LD files may be saved/transferred to various combinations of memory/storage medium, devices or systems and may also be transmitted using various transmission devices or systems. Consequently, some of blocks 410 to 420 may be modified or omitted.
• the color of the screen 18 and the neck and torso bib 20 may vary depending on the Chroma keying process used and that the disclosed use of the green color is a standard Chroma keying color given as a working example.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Physics & Mathematics (AREA)
• Acoustics & Sound (AREA)
• Signal Processing (AREA)
• General Engineering & Computer Science (AREA)
• Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
• Television Signal Processing For Recording (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=48798445

Family Applications (1)

Country Status (3)

Cited By (1)

Citations (4)

Family Cites Families (6)

• 2013
  • 2013-01-21 WO PCT/CA2013/000048 patent/WO2013106916A1/fr not_active Ceased
  • 2013-01-21 EP EP13738424.4A patent/EP2805483A4/fr not_active Withdrawn
• 2014
  • 2014-08-19 IN IN6918DEN2014 patent/IN2014DN06918A/en unknown

Patent Citations (4)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events