Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T17/00—Three-dimensional [3D] modelling for computer graphics
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
  • G06T19/20—Editing of three-dimensional [3D] images, e.g. changing shapes or colours, aligning objects or positioning parts
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
  • G06T2219/20—Indexing scheme for editing of 3D models
  • G06T2219/2012—Colour editing, changing, or manipulating; Use of colour codes
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
  • G06T2219/20—Indexing scheme for editing of 3D models
  • G06T2219/2021—Shape modification

Definitions

• FIG. 5 is a flowchart for an access-limiting process, in accordance with some examples.
• FIG. 7 illustrates a system for cross-modal shape and color manipulation, in accordance with some examples.
• FIG. 13 illustrates examples of cross-modal shape and color manipulations, in accordance with some examples.
• FIG. 21 is a diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies discussed herein, in accordance with some examples.
• the user can then edit the 3D shape 1 1904 by, for example, adding color to the 2D RGB view 1906.
• the editing module 902 uses an MM generator 701 to determine a second latent code that could be used to generate the 2D RGB view 1906 with the added color 1910 using the MM generator 701 with the second latent code as input.
• the editing module 902 then generates a new 3D shape 2 1908 using the second latent code.
• the MM generator 701 enables the editing module 902 to determine changes to a 3D shape 1 1904 based on changes made to a 2D sketch 1902 or a 2D RGB view 1906 and enables the editing module 902 to determine a 3D shape 1 1904 corresponding to a 2D sketch 1902 drawn by the user.
• the application servers 114 host a number of server applications and subsystems, including for example a messaging server 118, an image processing server 122, and a social network server 124.
• the messaging server 118 implements a number of message processing technologies and functions, particularly related to the aggregation and other processing of content (e.g., textual and multimedia content) included in messages received from multiple instances of the messaging client 104.
• content e.g., textual and multimedia content
• the text and media content from multiple sources may be aggregated into collections of content (e.g., called stories or galleries). These collections are then made available to the messaging client 104.
• Other processor and memory intensive processing of data may also be performed server-side by the messaging server 118, in view of the hardware requirements for such processing.
• an applet may incorporate a scripting language (e.g., a ,*js file or a .json file) and a style sheet (e.g., a ,*ss file).
• a scripting language e.g., a ,*js file or a .json file
• style sheet e.g., a ,*ss file
• FIG. 2 is a block diagram illustrating further details regarding the messaging system 100, according to some examples.
• the messaging system 100 is shown to comprise the messaging client 104 and the application servers 114.
• the messaging system 100 embodies a number of subsystems, which are supported on the client-side by the messaging client 104 and on the server-side by the application servers 114. These subsystems include, for example, an ephemeral timer system 202, a collection management system 204, an augmentation system 208, a map system 210, a game system 212, an external resource system 214, and a cross-modal shape and color manipulation system 216.
• the ephemeral timer system 202 is responsible for enforcing the temporary or time-limited access to content by the messaging client 104 and the messaging server 118.
• the ephemeral timer system 202 incorporates a number of timers that, based on duration and display parameters associated with a message, or collection of messages (e.g., a story), selectively enable access (e.g., for presentation and display) to messages and associated content via the messaging client 104. Further details regarding the operation of the ephemeral timer system 202 are provided below.
• the collection management system 204 is responsible for managing sets or collections of media (e.g., collections of text, image video, and audio data).
• a collection of content e.g., messages, including images, video, text, and audio
• Such a collection may be made available for a specified time period, such as the duration of an event to which the content relates. For example, content relating to a music concert may be made available as a “story” for the duration of that music concert.
• the collection management system 204 may also be responsible for publishing an icon that provides notification of the existence of a particular collection to the user interface of the messaging client 104.
• the augmentation system 208 uses the geolocation of the client device 102 to identify a media overlay that includes the name of a merchant at the geolocation of the client device 102.
• the media overlay may include other indicia associated with the merchant.
• the media overlays may be stored in the database 126 and accessed through the database server 120.
• the augmentation system 208 provides a merchant-based publication platform that enables merchants to select a particular media overlay associated with a geolocation via a bidding process. For example, the augmentation system 208 associates the media overlay of the highest bidding merchant with a corresponding geolocation for a predefined amount of time.
• the map system 210 provides various geographic location functions and supports the presentation of map-based media content and messages by the messaging client 104.
• the map system 210 enables the display of user icons or avatars (e.g., stored in profile data 316) on a map to indicate a current or past location of "friends" of a user, as well as media content (e.g., collections of messages including photographs and videos) generated by such friends, within the context of a map.
• a message posted by a user to the messaging system 100 from a specific geographic location may be displayed within the context of a map at that particular location to “friends” of a specific user on a map interface of the messaging client 104.
• An entity table 306 stores entity data, and is linked (e.g., referentially) to an entity graph 308 and profile data 316. Entities for which records are maintained within the entity table 306 may include individuals, corporate entities, organizations, objects, places, events, and so forth. Regardless of entity type, any entity regarding which the messaging server system 108 stores data may be a recognized entity. Each entity is provided with a unique identifier, as well as an entity type identifier (not shown).
• the profile data 316 stores multiple types of profile data about a particular entity.
• the profile data 316 may be selectively used and presented to other users of the messaging system 100, based on privacy settings specified by a particular entity.
• the profile data 316 includes, for example, a username, telephone number, address, settings (e.g., notification and privacy settings), as well as a user-selected avatar representation (or collection of such avatar representations).
• a particular user may then selectively include one or more of these avatar representations within the content of messages communicated via the messaging system 100, and on map interfaces displayed by messaging clients 104 to other users.
• the collection of avatar representations may include “status avatars,” which present a graphical representation of a status or activity that the user may select to communicate at a particular time.
• Filters are overlays that are displayed as overlaid on an image or video during presentation to a recipient user. Filters may be of various types, including user-selected filters from a set of filters presented to a sending user by the messaging client 104 when the sending user is composing a message. Other types of filters include geolocation filters (also known as geo-filters), which may be presented to a sending user based on geographic location. For example, geolocation filters specific to a neighborhood or special location may be presented within a user interface by the messaging client 104, based on geolocation information determined by a Global Positioning System (GPS) unit of the client device 102.
• GPS Global Positioning System
• multiple augmented reality content items that apply different pseudorandom movement models can be applied to the same content by selecting different augmented reality content items for the content.
• real-time video capture may be used with an illustrated modification to show how video images currently being captured by sensors of a client device 102 would modify the captured data. Such data may simply be displayed on the screen and not stored in memory, or the content captured by the device sensors may be recorded and stored in memory with or without the modifications (or both).
• a preview feature can show how different augmented reality content items will look within different windows in a display at the same time. This can, for example, enable multiple windows with different pseudorandom animations to be viewed on a display at the same time.
• neural network analysis of video frames may be used to place images, models, or textures in content (e.g., images or frames of video).
• Augmented reality content items thus refer both to the images, models, and textures used to create transformations in content, as well as to additional modeling and analysis information needed to achieve such transformations with object detection, tracking, and placement.
• elements to be transformed are identified by the computing device, and then detected and tracked if they are present in the frames of the video.
• the elements of the object are modified according to the request for modification, thus transforming the frames of the video stream. Transformation of frames of a video stream can be performed by different methods for different kinds of transformation. For example, for transformations of frames mostly referring to changing forms of object's elements characteristic points for each element of an object are calculated (e.g., using an Active Shape Model (ASM) or other known methods). Then, a mesh based on the characteristic points is generated for each of the at least one element of the object. This mesh is used in the following stage of tracking the elements of the object in the video stream.
• ASM Active Shape Model
• the transform system initiates a process to convert the image of the user to reflect the selected modification icon (e.g., generate a smiling face on the user).
• a modified image or video stream may be presented in a graphical user interface displayed on the client device 102 as soon as the image or video stream is captured, and a specified modification is selected.
• the transformation system may implement a complex convolutional neural network on a portion of the image or video stream to generate and apply the selected modification. That is, the user may capture the image or video stream and be presented with a modified result in real-time or near real-time once a modification icon has been selected. Further, the modification may be persistent while the video stream is being captured, and the selected modification icon remains toggled. Machine-taught neural networks may be used to enable such modifications.
• a story table 314 stores data regarding collections of messages and associated image, video, or audio data, which are compiled into a collection (e.g., a story or a gallery).
• the creation of a particular collection may be initiated by a particular user (e.g., each user for which a record is maintained in the entity table 306).
• a user may create a “personal story” in the form of a collection of content that has been created and sent/broadcasted by that user.
• the user interface of the messaging client 104 may include an icon that is user-selectable to enable a sending user to add specific content to his or her personal story.
• a collection may also constitute a “live story,” which is a collection of content from multiple users that is created manually, automatically, or using a combination of manual and automatic techniques.
• a “live story” may constitute a curated stream of user-submitted content from various locations and events. Users whose client devices have location services enabled and are at a common location event at a particular time may, for example, be presented with an option, via a user interface of the messaging client 104, to contribute content to a particular live story. The live story may be identified to the user by the messaging client 104, based on his or her location. The end result is a “live story” told from a community perspective.
• a further type of content collection is known as a “location story,” which enables a user whose client device 102 is located within a specific geographic location (e.g., on a college or university campus) to contribute to a particular collection.
• a contribution to a location story may require a second degree of authentication to verify that the end-user belongs to a specific organization or other entity (e.g., is a student on the university campus).
• FIG. 4 is a schematic diagram illustrating a structure of a message 400, according to some examples, generated by a messaging client 104 for communication to a further messaging client 104 or the messaging server 118.
• the content of a particular message 400 is used to populate the message table 302 stored within the database 126, accessible by the messaging server 118.
• the content of a message 400 is stored in memory as “in-transit” or “in-flight” data of the client device 102 or the application servers 114.
• a message 400 is shown to include the following example components:
• message identifier 402 a unique identifier that identifies the message 400.
• message text payload 404 text, to be generated by a user via a user interface of the client device 102, and that is included in the message 400.
• message duration parameter 414 parameter value indicating, in seconds, the amount of time for which content of the message (e.g., the message image payload 406, message video payload 408, message audio payload 410) is to be presented or made accessible to a user via the messaging client 104.
• each message 400 may be tagged with multiple tags, each of which is indicative of the subject matter of content included in the message payload. For example, where a particular image included in the message image payload 406 depicts an animal (e.g., a lion), a tag value may be included within the message tag 420 that is indicative of the relevant animal. Tag values may be generated manually, based on user input, or may be automatically generated using, for example, image recognition.
• the contents (e.g., values) of the various components of message 400 may be pointers to locations in tables within which content data values are stored.
• an image value in the message image payload 406 may be a pointer to (or address of) a location within an image table 312.
• values within the message video payload 408 may point to data stored within a video table 304
• values stored within the message augmentations 412 may point to data stored in an augmentation table 310
• values stored within the message story identifier 418 may point to data stored in a story table 314, and values stored within the message sender identifier 422 and the message receiver identifier 424 may point to user records stored within an entity table 306.
• FIG. 5 is a schematic diagram illustrating an access-limiting process 500, in terms of which access to content (e.g., an ephemeral message 502, and associated multimedia payload of data) or a content collection (e.g., an ephemeral message group 504) may be time-limited (e.g., made ephemeral).
• content e.g., an ephemeral message 502, and associated multimedia payload of data
• a content collection e.g., an ephemeral message group 504
• time-limited e.g., made ephemeral
• the group timer 514 operationally controls the overall lifespan of an associated ephemeral message group 504, as well as an individual ephemeral message 502 included in the ephemeral message group 504.
• each and every ephemeral message 502 within the ephemeral message group 504 remains viewable and accessible for a time period specified by the group duration parameter 508.
• a certain ephemeral message 502 may expire, within the context of ephemeral message group 504, based on a group participation parameter 512.
• a message duration parameter 506 may still determine the duration of time for which a particular ephemeral message 502 is displayed to a receiving user, even within the context of the ephemeral message group 504. Accordingly, the message duration parameter 506 determines the duration of time that a particular ephemeral message 502 is displayed to a receiving user, regardless of whether the receiving user is viewing that ephemeral message 502 inside or outside the context of an ephemeral message group 504.
• FIG. 6 illustrates a system 600 for cross-modal shape and color manipulation, in accordance with some examples.
• the system 600 takes a value, z, from latent space (Z) 602 and uses z as input to multi-modal (MM) generator 604 to generate 606, 610, 612, a three-dimensional (3D) shape 608, 2D RGB views 616, and a two-dimensional (2D) sketch 614, respectively.
• MM multi-modal
• the 3D shape 608, 2D RGB views 616, and 2D sketch 614 represent a same object, which in this case is an airplane.
• SDFs signed distance fields
• the term z R 722 indicates that the kth layer of 730 is fed into fp 728 as input.
• the system 700 combines the SDFs 720 with the 3D color 718 to generate or render the 3D shape 608 by Q e 607, which represents the generation of the 3D shape 608.
• the system 700 indicates either the training module 810 or the editing module 902.
• the system 700 generates the 3D shape 608 based on Equation (4).
• the system 700 determines the 2D sketches 716 based on Equation (5).
• the system 700 determines the 2D RGB views 714 based on Equation (6).
• the 2D RGB views 714 are termed 2D color views, in accordance with some examples.
• Equation(7) V (( , 0
• x) are parameters of the MM encoder 824, 0 are parameters of the MM generator 701, x is an observation variable, KL is the Kullback-Leibler divergence, z is a latent variable, p is the distribution, and E is an expectation.
• x) approximates the posterior distribution p(z
• the training module 810 estimates the parameters of the MM encoder 824 by maximizing Equation (9) based on Equation (7) while the parameters 6 of MM generator 701 are frozen. Equation (9): 0,Xj ).
• the training module 810 learns three modalities x, w, and y where
• the training module 810 uses as an objective function for training Q c 607 of Z 15 which is a measure of the loss between the input 3D shape 816 to the MM encoder 824 and the output 3D shape 804 from the MM generator 701.
• the training module 810 uses as an objective function for training Q s 712 of £ s , which is a measure of the crossentropy loss between input 2D sketch 820 to the MM encoder 824 and the output 2D sketch 808 from the MM generator 701.
• the training module 810 uses as an objective function for training 710 of L R , which is given by Equation (12).
• Equation (17): z arg may be determined based on Equation (16).
• the editing module 902 iterates to find z and determines that z is to be used as the latent code for the 2D sketch 908 based on a difference between Q M (z) and x M being less than a first threshold or not transgressing the first threshold and/or Z REG (z) being less than a second threshold or not transgressing the second threshold.
• FIG. 15 illustrates examples 1500 of cross-modal shape and color manipulations, in accordance with some examples.
• Column A 1502 is before a user adds any colors.
• the columns B 1504, C 1506, D 1508, and E 1510 have edits where color is added to the 2D RGB view 1520, at colored 1512, 1514, 1516, and 1518, respectively, and the 2D RGB view 1524, at colored 1528, 1530, 1532, and 1534, respectively.
• the row 3D shapes 1522 illustrates the shapes generated by the editing module 902 based on the latent code z of the corresponding shape of the row of 2D RGB views 1520.
• FIG. 16 illustrates examples 1600 of cross-modal shape and color manipulations, in accordance with some examples.
• the 2D RGB views 1608 are provided for a user by the editing module 902.
• the 3D shapes 1610 are generated by the editing module 902 based on the latent space Z 602 and the 2D RGB views 1608 with the edits of colored 1612, 1615, 1616, 1618 1620, and 1622 as described herein.
• a user colored 1612 the seat bottom of a chair that was a striped color overall and the resulting 3D shape 1610 in column A 1602 is that the entire chair has been colored with the color of colored 1612.
• the method 2000 continues at operation 2006 with generating a 3D shape from the latent code based on a second VAD with the latent code as input.
• the editing module 902 generates 3D shape 912 using the MM generator 701 at Q c 607 with the latent code as input.
• the method 2000 continues at operation 2008 with displaying the 3D shape on a display of a computing device.
• the editing module 902 may display 3D shape 1 1904 on a display 906.
• FIG. 21 is a diagrammatic representation of the machine 2100 within which instructions 2110 (e.g., software, a program, an application, an applet, an app, or other executable code) for causing the machine 2100 to perform any one or more of the methodologies discussed herein may be executed.
• the instructions 2110 may cause the machine 2100 to execute any one or more of the methods described herein.
• the instructions 2110 transform the general, non-programmed machine 2100 into a particular machine 2100 programmed to carry out the described and illustrated functions in the manner described.
• the machine 2100 may operate as a standalone device or may be coupled (e.g., networked) to other machines.
• the environmental components 2134 include, for example, one or cameras (with still image/photograph and video capabilities), illumination sensor components (e.g., photometer), temperature sensor components (e.g., one or more thermometers that detect ambient temperature), humidity sensor components, pressure sensor components (e.g., barometer), acoustic sensor components (e.g., one or more microphones that detect background noise), proximity sensor components (e.g., infrared sensors that detect nearby objects), gas sensors (e.g., gas detection sensors to detection concentrations of hazardous gases for safety or to measure pollutants in the atmosphere), or other components that may provide indications, measurements, or signals corresponding to a surrounding physical environment.
• illumination sensor components e.g., photometer
• temperature sensor components e.g., one or more thermometers that detect ambient temperature
• humidity sensor components e.g., pressure sensor components (e.g., barometer)
• acoustic sensor components e.g., one or more microphones that detect background noise
• proximity sensor components e.
• the operating system 2212 manages hardware resources and provides common services.
• the operating system 2212 includes, for example, a kernel 2214, services 2216, and drivers 2222.
• the kernel 2214 acts as an abstraction layer between the hardware and the other software layers.
• the kernel 2214 provides memory management, processor management (e.g., scheduling), component management, networking, and security settings, among other functionality.
• the services 2216 can provide other common services for the other software layers.
• the drivers 2222 are responsible for controlling or interfacing with the underlying hardware.
• the drivers 2222 can include display drivers, camera drivers, BLUETOOTH® or BLUETOOTH® Low Energy drivers, flash memory drivers, serial communication drivers (e.g., USB drivers), WI-FI® drivers, audio drivers, power management drivers, and so forth.
• the libraries 2210 provide a common low-level infrastructure used by the applications 2206.
• the libraries 2210 can include system libraries 2218 (e.g., C standard library) that provide functions such as memory allocation functions, string manipulation functions, mathematic functions, and the like.
• the third- party application 2240 e.g., an application developed using the ANDROIDTM or IOSTM software development kit (SDK) by an entity other than the vendor of the particular platform
• the third-party application 2240 may be mobile software running on a mobile operating system such as IOSTM, ANDROIDTM, WINDOWS® Phone, or another mobile operating system.
• the third-party application 2240 can invoke the API calls 2250 provided by the operating system 2212 to facilitate functionality described herein.
• Modules refers to any logic, components, modules, or mechanisms. Modules may constitute either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules.
• a “hardware module” is a tangible unit capable of performing certain operations and may be configured or arranged in a certain physical manner.
• one or more computer systems e.g., a standalone computer system, a client computer system, or a server computer system
• one or more hardware modules of a computer system e.g., a processor or a group of processors
• software e.g., an application or application portion
• Communication network refers to one or more portions of a network that may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of the Public Switched Telephone Network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, a Wi-Fi® network, another type of network, or a combination of two or more such networks.
• VPN virtual private network
• LAN local area network
• WLAN wireless LAN
• WAN wide area network
• WWAN wireless WAN
• MAN metropolitan area network
• PSTN Public Switched Telephone Network
• POTS plain old telephone service
• the phrase "hardware component”(or “hardware-implemented component”) should be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein.
• hardware components are temporarily configured (e.g., programmed)
• each of the hardware components need not be configured or instantiated at any one instance in time.
• a hardware component comprises a general -purpose processor configured by software to become a special-purpose processor
• the general -purpose processor may be configured as respectively different special-purpose processors (e.g., comprising different hardware components) at different times.
• Computer-readable storage medium refers to both machine-storage media and transmission media. Thus, the terms include both storage devices/media and carrier waves/modulated data signals.
• machine-readable medium “computer- readable medium” and “device-readable medium” mean the same thing and may be used interchangeably in this disclosure.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Computer Graphics (AREA)
• Software Systems (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Geometry (AREA)
• Architecture (AREA)
• Computer Hardware Design (AREA)
• General Engineering & Computer Science (AREA)
• Information Transfer Between Computers (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=86764417

Family Applications (1)

Country Status (4)

Families Citing this family (1)

Family Cites Families (3)

• 2023
  • 2023-05-08 EP EP23730620.4A patent/EP4533406A1/de active Pending
  • 2023-05-08 KR KR1020247042277A patent/KR20250012132A/ko active Pending
  • 2023-05-08 CN CN202380043918.1A patent/CN119452396A/zh active Pending
  • 2023-05-08 WO PCT/US2023/021311 patent/WO2023235110A1/en not_active Ceased

Also Published As

Similar Documents

Legal Events