WO2008024486A2 - Beat and text based editing and composing systems and methods - Google Patents

Abstract

An audiovisual composing tool for enhancing audiovisual work creation and editing can include a graphical user interface (GUI) that displays lyrics of selected songs, mapped to a beat/tempo grid, on a time line. The composing tool can allow a user to find pictures or video clips stored on the user's computer, for example, and drag and drop the pictures and video clips into a time line window. When dropped into the time line, the pictures or videos can automatically snap to the beat/tempo grid, so that when played back as a complete audiovisual work, the visual elements (e.g. images and video) display in synchronization to the audio of the work. Alternatively, instead of displaying a time line with beat markers, a wizard tool can be provided that automatically synchronizes photos or videos to the beats and/or lyrics of a song. Once satisfied with the audiovisual work, a user can render the work into a media file and share the media file with others.

Description

Beat and Text Based Editing and Composing Systems and Methods

Cross Reference to Related Applications

[0001] This application claims the benefit of priority from U.S. Provisional Application No. 60/919,626 filed on March 22, 2007, the full disclosure of which is incorporated by reference for all purposes.

[0002] This application also claims the benefit of priority from U.S.

Provisional Application No. 60/840,240 filed August 24, 2006, the full disclosure of which is incorporated by reference for all purposes. Field of the Invention

[0003] This relates generally to systems and methods for composing audiovisual works, and, in particular, a system and method that maps transitions in audiovisual works to lyrics and beats of music in the work.

Background of the Invention [0004] Recently, use of audiovisual works or presentations have become very popular. One reason for this is the growth of social networking sites that enable users to post audiovisual works on the sites. The audiovisual works can include photos, videos and music, for example. Such sites include My Space, Friendster, YouTube and others. [0005] However, composing and editing audiovisual works can be complicated and time consuming. Moreover, it is commonly desired to synchronize presentation of photos and videos in the audiovisual works with the music, which can be very difficult using conventional audiovisual composing and editing methods.

Summary of the Invention [0006] As will be described in greater detail below, embodiments of the present invention can include features for enhancing audiovisual work creation and editing. In accordance with various embodiments, a user can install software onto the user's personal computer or launch a web-based version of an audiovisual presentation composing tool. Once software is installed or the user is logged into a web-based version, the user can select one or more songs to use in an audiovisual work. A graphical user interface (GUI) of the composing tool can then display the lyrics of the selected songs, mapped to a beat/tempo grid, on a timeline. The composing tool can allow a user to then find pictures or video clips stored on the user's computer, for example, and drag and drop the pictures and video clips into a timeline window. When dropped into the timeline, the pictures or videos can automatically snap to the beat/tempo grid, so that when played back as a complete audiovisual work, the visual elements (e.g. photos and video) display in synchronization to the audio portion of the work. Alternatively, instead of displaying a timeline with beat markers, a wizard tool can be provided that automatically synchronizes the visual elements to the beats and/or lyrics of a song. Once satisfied with the audiovisual work, a user can render the work into a media file and share the media file with others.

Brief Description of the Drawings [0007] FIG. 1 is simplified view of an exemplary client-server environment in accordance with an embodiment of the present invention. [0008] FIG. 2 is an example of a computer system upon which one embodiment of the present invention can be implemented. [0009] FIG. 3 illustrates an exemplary graphical user interface in accordance with one embodiment of the present invention.

[0010] FIG. 4 illustrates an exemplary music selection page in accordance with one embodiment of the invention.

[0011] FIG. 5 illustrates exemplary Add Photos page in accordance with one embodiment of the invention. [0012] FIG. 6 illustrates exemplary effects pop up window in accordance with one embodiment of the invention.

[0013] FIG. 7 illustrates exemplary Make Video page in accordance with one embodiment of the invention.

Detailed Description of the Preferred Embodiment [0014] A method and system for composing and editing audiovisual presentations are described. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced without some of these specific details. In other instances, well-known structures and devices are shown in block diagram form.

(0015] Exemplary embodiments of the present invention can be embodied in machine-executable instructions. The instructions can be used to cause a general- purpose or special-purpose processor which is programmed with the instructions to perform the steps. Alternatively, the steps can be performed by specific hardware components that contain hardwired logic for performing the steps, or by any combination of programmed computer components and custom hardware components. [0016] Embodiments of the present invention can be provided as a computer program product which may include a machine-readable medium having stored thereon instructions which can be used to program a computer (or other electronic devices) to perform a process according to embodiments of the present invention. The machine-readable medium can include, but is not limited to, floppy diskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, magnet or optical cards, or other type of media machine-readable medium suitable for storing electronic instructions. Moreover, embodiments of the present invention can also be downloaded as a computer program product, wherein the program may be transferred from a remote computer (e.g., a server) to a requesting computer (e.g., a client) by way of data signals embodied in a carrier wave or other propagation medium via a communication link (e.g., a modem or network connection). [0017] FIG. 1 is a simplified view of an exemplary client-server environment, such as the World Wide Web (the Web), in which online audiovisual editing and composing can take place. The architecture of the Web follows a conventional client- server model. The terms "client" and "server" are used to refer to a computer's general role as a requester of data (the client) or provider of data (the server). Web clients 105 and Web servers 110 can communicate using a protocol such as HyperText Transfer Protocol (HTTP). Flash technology, developed by Macromedia of San Francisco, California, can also be used for adding animation and interactivity to web pages. In the Web environment, Web browsers can reside on clients and render Web document (pages) served by the Web servers. The client-server model can be used to communicate information between clients 105 and servers 1 10. Web servers 1 10 can be coupled to network or communication medium 100, such as the Internet, and respond to document requests and/or other queries from Web clients 105. When a user selects a document by submitting its Uniform Resource Locator (URL), a Web browser, such as Internet Explorer, Firefox or Safari, can open a connection to a server 1 10 and initiate a request (e.g., an HTTP get) for the document. The server 110 can deliver the requested document, typically in the form of a text document coded in a standard markup language such as HyperText Markup Language (HTML). [0018] In an exemplary embodiment, server (provider) 110 can provide support to encode audiovisual works based on Extensible Markup Language (XML) strings passed from a Flash platform to an audiovisual generation tool. The provider 110 can also store the audiovisual works and images in a mass storage memory device.

[0019] A computer system 200 representing an exemplary client computer in which features of the present invention may be implemented will now be described with reference to FIG. 2. The computing system 200 can represent, for example, a desktop, laptop, and notebook computers; hand held computing devices (PDA's, cell phones, palmtops, etc.); mainframes, supercomputers, or servers; or any other type of special or general purpose computing devices as may be desirable or appropriate for a given application or environment.

[0020] The computer system 200 can include a bus or other communication means 201 for communicating information, and a processing means such as processor 202 coupled with bus 201 for processing information. Computer system 200 can further include a random access memory (RAM) or other dynamic storage device 204 (referred to as main memory), coupled to bus 201 for storing information and instructions to be executed by processor 202. Main memory 204 also may be used for storing temporary variables or other intermediate information during execution of instructions by processor 202. Computer system 200 can also have a read only memory (ROM) and/or other static storage device 206 coupled to bus 201 for storing static information and instructions for processor 202.

[0021] A data storage device 207 such as a magnetic disk or optical disc and its corresponding drive can also be coupled to computer system 200 for storing information and instructions. Computer system 200 can be coupled via bus 201 to a display device 221, such as a cathode ray tube (CRT) or Liquid Crystal Display (LCD), for displaying information to a computer user. Typically, an alphanumeric input device 222, including alphanumeric and other keys, may be coupled to bus 201 for communicating information and/or command selections to processor 202. Another type of user input device is cursor control 223, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections to processor 202 and for controlling cursor movement on display 221. [0022] In addition, a communication device 225 can be coupled to bus 201 for accessing remote servers via the Internet, for example. The communication device 225 can include a modem, a network interface card, or other commercially available network interface devices, such as those used for coupling to an Ethernet, token ring, or other type of network. In any event, in this manner, the computer system 200 can be coupled to a number of servers and/or other clients via a conventional network infrastructure, such as a company's Intranet and/or the Internet, for example.

[0023] FIG. 3 illustrates a main user graphical interface 300 (herein after

"GUI 300") of an audiovisual editing tool on computer 200 in accordance with an exemplary embodiment of the present invention. The GUI 300 can be partitioned into a plurality of display areas, for example, two parts in this embodiment. As shown generally in FIG. 3, an editing window 302 and an explorer window 304 can be displayed at predetermined positions on the GUI 300. The audiovisual editing tool can be stored locally on a computer device, or can be hosted by a provider and accessed via a communication medium, such as the Internet. [0024] In general, the explorer window 304 can serve as an interface to allow a user to find songs, videos, photos or other media (hereinafter collectively referred to as "visual elements") to be used in an audiovisual work. In an exemplary embodiment, a user can access both local databases and remote databases connected to computer 200 to search for visual elements to include in an audiovisual work. In the embodiment shown in FIG. 3, the explorer window includes music tab 306 and photos tab 308. When the music tab 306 is selected, the explorer window 304 can be used to access file systems containing music files. Accordingly, a user can use the explorer window 304 to navigate through file systems and select one or more music files that are to be used for composing an audiovisual presentation. Similarly, when the photos tab 306 is selected, a user can navigate through file systems to select visual elements to be used in the presentation. Selected visual elements can then be uploaded to a designated database, local or remote, and displayed in a preview pane 312 for later use. In the embodiment shown in FIG. 3, the preview pane 312 contains thumbnail images 310 of the visual elements. In accordance with an exemplary embodiment, navigating through file systems can be performed in a similar manner as with Windows Explorer®.

[0025] With further reference to FIG. 3, the editing window 302 can be used to compose and edit an audiovisual work using a "drag and drop" method. A user can select a desired thumbnail image 310 of a visual element from the preview pane 312 and drag and drop the thumbnail image 310 to a position of the user's preference along a timeline 314. The timeline 314 can include an incremented section 316 for providing a reference as to a timeline of the presentation and a media row 318 which can display the location of selected thumbnail images 310 along the timeline 314. Furthermore, a lyric row 320 can display lyrics of the audio portion of the presentation and a beat/tempo row 322 can display a waveform of the beat/tempo of the audio portion of the presentation. In accordance with various embodiments, the lyrics in lyric row 320 and the beat/tempo in the beat/tempo row 322 can be mapped to the timeline 314 so that a user can easily identify a location where a user desires to drop a thumbnail image 310. As a result, the user can use the timeline 314, lyric row 320 and beat/tempo row 322 as visual references to drop thumbnail images 310 in desired sequences in the audiovisual work.

[0026] The timeline 314 can also include a plurality of beat markers 324. The beat markers 324 can be positioned along the timeline 314 to have a desired relationship with the tempo/beat and lyrics of a song. Predetermined logic can be used to determine the positioning of the beat markers 324. In one embodiment, logic takes into account the length of a song, the number of visual elements used in the presentation, the theme of the presentation and the beat/tempo of the song. Logic can also dictate that beat markers 324 are placed at the closest beats that are located at least five seconds apart. In addition, logic can determine the location of the beats in an audio track by analyzing the location of waveform peaks in the audio track. Photos can also be repeated or looped in the audiovisual work if the number of visual elements used is less than the number of beat markers in the presentation. Alternatively, if more visual elements are used than the number of available beat markers 324, then the visual elements can be displayed as sequenced in the timeline 314 and can continue as the song repeats. Multiple songs can be used in the presentation, so the visual elements can begin anew at the location of the first beat marker 324 in each song.

[0027] As referenced above, a user can drag and drop thumbnail images 310 into the timeline 314 so that the associated visual elements appear in the time sequence in which they appear in the timeline 314. By doing so, a user can easily and efficiently construct the overall layout of an audiovisual work. When dropped into the timeline 314, the thumbnail images 310 can automatically snap to the closest beat marker 324 so that when played back as a complete audiovisual presentation, the visual elements display in synchronization to the audio of the presentation. For example, in the song "Winter Wonderland," a user might drop a photo of a snowman onto the timeline 314 at the point in time where a beat marker 324 occurs in close proximity to the visual display of the word "snowman" in the lyrics row 316. This allows a user to easily synchronize the display of visual elements with the lyrics of a song.

[0028] In accordance with various exemplary embodiments, when an audiovisual work is being played, a visual element can be displayed from the point in the timeline 314 where the beat marker 324 associated with the visual element is located until at the location of the following beat marker. The next visual element associated with the following beat marker 324 can then be displayed. With regards to video clips inserted into the presentation, a video clip can start playing at the location where its associated beat marker 324 is placed and can end at the location of the following beat marker. If the video clip is longer than the time between beat markers, then the video clip can be clipped so that the video ends at the closest beat marker that is shorter than the chosen clip length.

[0029] A user can preview the work in full or from any of the beat markers

320. In addition, at any point during the audiovisual work creation process, the audiovisual work can be encoded into various file formats, for example mpeg or avi format, saved, and exported to other computers over the network 100. Saved presentations can also be later reloaded for further editing.

[0030] It is appreciated that exemplary embodiments can also include production/authoring tools for karaoke, closed captioning, or any use where synchronization of visual elements to a given audio element can be useful. [0031] An exemplary embodiment of a wizard version of an audiovisual presentation creation system and method is described with reference to FIGS. 4-7. In general, a wizard version can automate many of the steps of creation and can also allow a user to add more detail. In the embodiment described in FIGS. 4-7, a template can be created that maps the beat of the audio portion (e.g., music) of the presentation. The template need not be displayed to a user; rather, the user need only select an order of visual elements to be inserted in the template. The wizard can then automatically map the visual elements to the template and display the photos in accordance with the beat of the music in the presentation. In other words, in contrast to the embodiment described in FIG. 3, the embodiment described with reference to FIGS. 4-7 can map the photos to beat markers automatically without a user needing to associate the thumbnail images 310 with particular beat markers 324. [0032] Software implementing the wizard version can be downloaded onto a user's computer, such as computer 200 or a user can launch a web site implementing the wizard version, for example. Embodiments of the wizard version can also guide a user through various setup pages, enabling a user to easily create an audiovisual work with visual elements synchronized to the beat of the audio in the work. [0033] FIG. 4 illustrates a Music Selection page 400 where a user selects music to be used in an audiovisual work. User selectable categories of music can be displayed in a category selection pane 402 and music corresponding to a selected category can be displayed in a music preview pane 404. A user can preview a song listed in the music preview pane 404 by selecting a preview button 406 associated with the listed song. A user can also select the songs of their choice to be used in the presentation by highlighting the songs and then selecting a "Next" button 408. [0034] The Music Selection page 400 can also permit a user to save a project by selecting a Save and Finish button 410. The user can then resume the presentation creation process at a later date by accessing the saved project. [0035] Once the user hits the "Next" button 408 in the music page 400, the user can be presented with an Add Photos page 500, an exemplary embodiment of which is shown in FIG. 5. Here, the user can import photos and/or video clips from local or remote sites. In this regard, an explorer pane 502 can be provided with a list of sites for searching for and visual elements. For example, the list can include user selectable links associated with user's computer or various web sites, such as Friendster®, My Space®, flickr®, and Photobucket®. The list can also include a selectable URL link that, when selected, permits the user to enter a URL of a site not listed in the explorer pane. Once located, the visual elements can be saved on a provider's server, such as server 110 in FIG. 1, and can be used at a later date or during the present audiovisual creation process. The imported visual elements can be displayed as thumbnails 504 in a media row 506. [0036] Each thumbnail 504 can include a plurality of selectable icons used for editing the associated photo or video. In the embodiment shown in FIG. 5, each thumbnail 504 can include a delete icon 508, a right arrow 510, a left arrow 512, a rotate icon 514 and an effects icon 516. Selection of the delete icon 508 can remove the thumbnail 504 from the media row 506 so that the associated media file is not used in the resulting audiovisual presentation. Selection of the right arrow 510 and left arrow 512 can move the associated thumbnail 504 one position to the right or left, respectively, in the media row 506. In accordance with an exemplary embodiment, the visual elements can be displayed in a resulting audiovisual work in the order the thumbnails 504 placed in the media row 506. Accordingly, changing the order of the thumbnails 504 displayed in the media row 504 can change the order in which the associated visual elements are displayed in the resulting audiovisual work. Selection of the rotate icon 514 can cause the associated photo to rotate 90 degrees. Furthermore, selection of the effects icon 516 can permit a user to add text, effects and transitions to the audiovisual presentation.

[0037] When the effects icon 516 is selected, an effects pop up window 600 can be displayed as illustrated in FIG. 6. The effects window 600 can include various editing options associated with the selected thumbnail. For example, the user can add text in a caption box 602. That text can then be displayed in the resulting audiovisual presentation when the associated visual element is displayed. The effects window

600 can also permit a user to apply pan, crop and zone effects to the associated visual element and apply various types of transitions when the associated visual elements is displayed in the resulting audiovisual presentation. Once a user selects the desired effects, the user can select an Apply button 604 and the effects are saved and the effects window 600 is taken down. In this manner, effects can be added to each thumbnail 504 in the media row 506.

[0038] Further to FIG. 5, at any point in time while in the Add Photos page

500, the user can preview the audiovisual presentation. The user need only select a preview button 520 in the preview pane 522 and the audiovisual work is played in its current form in the preview pane 522.

[0039] With further reference to FIG. 5, a user can also save the audiovisual presentation at any point to work on further at a later time. The audiovisual composing system can save the user's work as a project, which can include the user's information, source visual elements, and XML instructions. The user need only select the Save and Finish Later button 524 and the project can be saved to memory. The user can access the saved project at a later time and resume editing the presentation. In one embodiment, the project can be saved at a flash client and an "Akaline Project" associated with the saved project is created. The project can then be passed to the provider. Saved projects can then be available on a My Videos page (not shown) that can be accessed by the user from the provider's web site. When the user later selects the saved project from the My Videos page, a unique video identification can be passed to the flash client. The flash client can use the identification to retrieve the associated XML and visual elements, and build the video project. The project can also be applied to any media platform- including DVD, mobile devices and video game players. The XML can specify where the project will be played. [0040] Once a user is satisfied with the presentation, the user can select a

"Next" button 526 on the Add Photos page 500. Doing so can bring up a Make Video page 700 shown in FIG. 7. The Make Video page 700 can prompt the user for various types of information relating to the presentation, such as title, category, description, privacy and tag information. Once completed, the user can select a Make Video button 702, which results in the work being encoded into a video format and made available for distribution. The resulting audiovisual work can be encoded for web distribution, mobile devices and saving to memory. [0041] While this invention has been described in terms of several preferred embodiments, there are alterations, permutations, and equivalents, which fall within the scope of this invention. For example, the term "computer" does not necessarily mean any particular kind of device, combination of hardware and/or software, nor should it be considered restricted to either a multi purpose or single purpose device. It is also understood that one of ordinary skill in the art would able to design and implement, without undue experimentation, some or all of the components utilized by the method and system of the present invention as purely executable software, or as hardware components (e.g. ASICs, programmable logic devices or arrays, etc.), or as firmware, or as any combination of these implementations. As used herein, the term " module " refers to any one of these components or any combination of components for performing a specified function, wherein each component or combination of components may be constructed or created in accordance with any one of the above i mplementations . [0042] It should also be noted that there are many alternative ways of implementing the methods and apparatuses of the present invention. It is therefore intended that the following appended claims be interpreted as including all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.

Claims

WHAT IS CLAIMED IS:

1. A method for facilitating creation and editing of an audiovisual work, comprising: (a) selecting at least one audio track to be included in the audiovisual work;

(b) creating a beat map having at least one beat marker, the at least one beat marker corresponding to a beat in the audio track;

(c) selecting at least one visual element to be included in the audiovisual work; and

(d) associating the at least one visual element with the at least one beat marker so that when the audiovisual work is played, the visual element is displayed at a point in time corresponding to the location of the at least one beat marker in the beat map.

2. The method of claim 1 , wherein the visual element is a digital image.

3. The method of claim 1, wherein the visual element is a video clip.

4. The method of claim 1 , wherein the audio track corresponds to a song.

5. The method of claim 1, further comprising:

(e) accessing a remote provider site via a network from the mobile device;

(f) performing steps (a) and (c) from the mobile device;

(g) receiving an encoded version of the audiovisual work from the provider site via the network; and

(h) playing the encoded version of the audiovisual work on the mobile device.

6. The method of claim 1 , further comprising displaying a graphical user interface, the graphical user interface having a lyric row for displaying lyrics associated with the at least one audio track, beat/tempo row for displaying an audio waveform of the beat/tempo of the at least one audio track, and a timeline synchronized with the lyrics and the beat/tempo of the audio track.

7. The method of claim 6, wherein the graphical user interface further comprises a visual representation to the at least one beat marker located on the timeline.

8. The method of claim 7, wherein the associating step (d) further comprises dragging and dropping a thumbnail image associated with the at least one visual element in the timeline, wherein, upon dropping the thumbnail image in the timeline, the thumbnail associates itself with the at least one beat maker.

9. The method of claim 1, wherein steps (b) and (d) are performed automatically.

10. A computer program product comprising a computer readable medium having computer program logic recorded thereon for the computer program logic comprising: a first module configured to receive a selection of at least one audio track to be included in the audiovisual work; a second module configured to create a beat map having at least one beat marker, the at least one beat marker corresponding to a beat in the audio track; a third module configured to receive a selection of at least one visual element to be included in the audiovisual work; and a fourth module configured to associate the at least one visual element with the at least one beat marker so that when the audiovisual work is played, the visual element is displayed at a point in time corresponding to the location of the at least one beat marker in the beat map.

11. The computer program product of claim 10, further comprising: a fifth module configured to communicate with a mobile device via a communication network; and a sixth module configured to receive the at least one audio track and the at least one visual element selections from the mobile device via the communication network; a seventh configured to encode the audiovisual work into an audiovisual work program file that is compatible with the operating system of the mobile device.

12. The computer program product of claim 10, further comprising a fifth module configured to display a graphical user interface on a user's computer, the graphical user interface having a lyric row for displaying lyrics associated with the audio track, a beat/tempo row for displaying an audio waveform of the beat/tempo of the audio track, and a timeline synchronized with the lyrics and the beat/tempo of the audio track.

13. The computer program product of claim 12, wherein the graphical user interface further comprises visual indications of the at least one beat marker's location along the timeline.

14. A system for creating and editing an audiovisual work, comprising: (a) means for selecting at least one audio track to be included in the audiovisual work;

(b) means for creating a beat map having at least one beat marker, the at least one beat marker corresponding to a beat in the audio track;

(c) means for selecting at least one visual element to be included in the audiovisual work; and

(d) means for associating the at least one visual element with the at least one beat marker so that when the audiovisual work is played, the visual element is displayed at a point in time corresponding to the location of the at least one beat marker in the beat map.

15. The system of claim 14, further comprising means for saving and retrieving a partially completed audiovisual project.

16. The system of claim 14, further comprising means for displaying a graphical user interface that facilitates composing and editing of the audiovisual work.