WO2023246888A1 - 多媒体数据处理方法、装置和计算机可读存储介质 - Google Patents

多媒体数据处理方法、装置和计算机可读存储介质 Download PDF

Info

Publication number
WO2023246888A1
WO2023246888A1 PCT/CN2023/101786 CN2023101786W WO2023246888A1 WO 2023246888 A1 WO2023246888 A1 WO 2023246888A1 CN 2023101786 W CN2023101786 W CN 2023101786W WO 2023246888 A1 WO2023246888 A1 WO 2023246888A1
Authority
WO
WIPO (PCT)
Prior art keywords
feature data
text
preset
mapping relationship
expression
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/101786
Other languages
English (en)
French (fr)
Inventor
惠羿
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ZTE Corp
Original Assignee
ZTE Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ZTE Corp filed Critical ZTE Corp
Priority to EP23826533.4A priority Critical patent/EP4543012A4/en
Priority to US18/878,212 priority patent/US20250384605A1/en
Publication of WO2023246888A1 publication Critical patent/WO2023246888A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/233—Processing of audio elementary streams
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—Two-dimensional [2D] image generation
    • G06T11/60—Creating or editing images; Combining images with text
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/4402—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display
    • H04N21/440236—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving reformatting operations of video signals for household redistribution, storage or real-time display by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174—Facial expression recognition
    • G06V40/176—Dynamic expression
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/18—Speech classification or search using natural language modelling
    • G10L15/1822—Parsing for meaning understanding
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/26—Speech to text systems
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/57—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for processing of video signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23412—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
    • H04N21/234336—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by media transcoding, e.g. video is transformed into a slideshow of still pictures or audio is converted into text
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/235—Processing of additional data, e.g. scrambling of additional data or processing content descriptors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/435—Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/439—Processing of audio elementary streams
    • H04N21/4394—Processing of audio elementary streams involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
    • H04N21/44012—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving rendering scenes according to scene graphs, e.g. MPEG-4 scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/81—Monomedia components thereof
    • H04N21/8106—Monomedia components thereof involving special audio data, e.g. different tracks for different languages
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/221—Announcement of recognition results
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state

Definitions

  • the embodiments of the present application relate to but are not limited to the field of data processing technology, and in particular, to a multimedia data processing method, device and computer-readable storage medium.
  • Embodiments of the present application provide a multimedia data processing method, device and computer-readable storage medium.
  • embodiments of the present application provide a multimedia data processing method, which includes: obtaining audio streams and video streams of multimedia data; parsing the audio streams to obtain text feature data, and matching the described data according to a preset mapping relationship Text feature data to determine topic feature data; parse the video stream to obtain expression feature data, and compare and match the expression feature data according to a preset mapping relationship to determine the emotion index; based on the text feature data, the emotion index and The topic feature data renders the multimedia data.
  • embodiments of the present application provide a multimedia data processing device, including: an audio processing module configured to receive and parse the audio stream to obtain text feature data; a video processing module configured to receive and parse the video stream, Obtain the expression feature data; the mapping relationship module is configured to process the text feature data to obtain topic feature data, and process the expression feature data to obtain the emotional index; the rendering module is configured to combine the text feature data and the topic feature The data and the sentiment index are rendered with the video stream.
  • embodiments of the present application provide an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above mentioned steps are implemented.
  • embodiments of the present application provide a computer-readable storage medium, which stores A computer executable program is stored, and the computer executable program is used to cause the computer to execute the multimedia data processing method described in the first aspect above.
  • Figure 1 is a main flow chart of a multimedia data processing method provided by an embodiment of the present application.
  • Figure 2A is a sub-flow chart of a multimedia data processing method provided by an embodiment of the present application.
  • Figure 2B is a sub-flow chart for parsing an audio stream to obtain text feature data provided by an embodiment of the present application
  • Figure 3A is a sub-flow chart of a multimedia data processing method provided by an embodiment of the present application.
  • Figure 3B is a schematic diagram of a micro-expression provided by an embodiment of the present application.
  • Figure 3C is a sub-flow chart for parsing a video stream to obtain expression feature data provided by an embodiment of the present application
  • Figure 3D is a schematic diagram of the mapping relationship between preset micro-expression combinations and emotional indexes provided by an embodiment of the present application
  • Figure 4A is a sub-flow chart of a multimedia data processing method provided by an embodiment of the present application.
  • Figure 4B is a schematic diagram of the mapping relationship between preset key phrases and scene templates provided by an embodiment of the present application.
  • Figure 4C is a sub-flow chart of a multimedia data processing method provided by an embodiment of the present application.
  • Figure 4D is a schematic diagram of the mapping relationship between preset declaration statements and command sequences provided by an embodiment of the present application.
  • Figure 4E is a schematic diagram of a command sequence type provided by an embodiment of the present application.
  • Figure 5A is a sub-flow chart of a multimedia data processing method provided by an embodiment of the present application.
  • Figure 5B is a schematic diagram of the mapping relationship between preset sensitive statements and rendering sets provided by an embodiment of the present application.
  • Figure 6 is a schematic diagram of the structure of a multimedia data processing device provided by an embodiment of the present application.
  • Figure 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
  • embodiments of the present application provide a multimedia data processing method, device and computer-readable storage medium.
  • obtain the audio stream and video stream of the multimedia data analyze the audio stream , obtain the text feature data, and compare and match the text feature data according to the preset mapping relationship to determine the topic feature data; parse the video stream to obtain the expression feature data, and compare and match the expression feature data according to the preset mapping relationship to determine the emotional index; based on the text Feature data, sentiment index and topic feature data render multimedia data.
  • this application analyzes and extracts, intelligently analyzes and renders the multimedia information of multi-person discussion calls, and digs out multi-dimensional value information based on the voice and video of existing call products, including voice text, emotion Index, and the content of the voice text is further processed according to the preset mapping relationship, so that the call product can present richer and smarter information.
  • this application can flexibly handle different business scenarios and talks.
  • Content expand the application fields of technical solutions, can effectively tap the potential value of call products, bring a new experience to call negotiations, increase the negotiation competitiveness of negotiation business users, improve decision-making efficiency, and win more voice. Seize business opportunities effectively.
  • FIG. 1 is a flow chart of a multimedia data processing method provided by an embodiment of the present application.
  • Multimedia data processing methods include but are not limited to the following steps:
  • Step S100 obtain the audio stream and video stream of multimedia data
  • Step S200 parse the audio stream to obtain text feature data, and compare and match the text feature data according to the preset mapping relationship to determine the topic feature data;
  • Step S300 parse the video stream to obtain expression feature data, and compare and match the expression feature data according to the preset mapping relationship to determine the emotional index;
  • Step S400 Render multimedia data based on text feature data, emotion index and topic feature data.
  • the multimedia transmitted in the network includes audio and video.
  • the audio stream and video of the multimedia data are first obtained.
  • the sounds in the audio stream are parsed and extracted to obtain text feature data.
  • the sounds in the audio stream can be human voices, and the human voices are converted into text, that is, language characters, through speech recognition technology; similarly, it can also be Whether it is a non-human voice, based on the frequency spectrum, identify its corresponding sound type, name, characteristics, etc., and convert the name or action corresponding to the sound into a text description.
  • the sound of an instrument is recognized as piano music, text features
  • the data can include piano music, the name of the piano music, related information of the piano music, etc.; then the video stream in the multimedia data is parsed to obtain expression feature data.
  • the expression feature data includes the expressions of the participants in the video conversation, for example, through the video Recognize consecutive frames in the stream, analyze the faces in the video, extract the dynamic changes in facial expressions, and generate different expression feature data for different expressions. For example, identify the angry expression on the face through the eyebrows. , the combination of eyes, face, nose, lips and other postures generates expression feature data.
  • the text feature data is matched according to the preset mapping relationship to determine the topic feature data.
  • the preset mapping relationship is to set some text feature data and topic feature data into a mapping relationship, that is, some text short sentences are followed by words.
  • the question keyword is set as a mapping relationship.
  • the text phrase and some action commands can also be set as a mapping relationship.
  • the text phrase is matched as a search action, that is, if a match is achieved, the matched phrase is retrieved. , use the search results as topic feature data.
  • the text feature data is obtained after multi-level mapping transformation and action processing. Therefore, according to The preset mapping relationship matches the text phrases in the text feature data with the preset text phrases in the mapping relationship. If the matching degree reaches the preset threshold, the text phrases in the text feature data are matched according to the preset mapping relationship.
  • the keywords mapped to the preset text phrases matched by the text phrases are determined as topic feature data.
  • the text feature data may contain an entire text.
  • the text in the text feature data is dispersed into short sentences or short words, and the scattered short sentences or short words are combined with the preset
  • the preset threshold can be understood as the proportion of short sentences or short words scattered in the text feature data that match the preset short text sentences. You can gradually test by setting values with different granularities. A suitable preset threshold range can also be used to determine whether the threshold setting is appropriate through the recognition effect.
  • the preset mapping relationship is to set some expression feature data and the emotion index into a mapping relationship, that is, to set the expression extracted from the video stream and the emotion index.
  • the emotional index can be expressed in the form of color, numerical value, graphics, etc.
  • the emotion is set from angry to happy to 0-100 points, 0 means very angry, and 100 means very happy.
  • the expression extracted from the video stream can be matched to the corresponding score.
  • the emotional index can be represented by color. Different colors correspond to different emotions. Through color psychology, warm colors represent boldness. , sunshine, enthusiasm, enthusiasm, and liveliness. Cold colors represent gracefulness, femininity, calmness, and elegance.
  • Expression feature data can include expression changes over a period of time, and may include multiple expression combinations.
  • the preset threshold can be understood as comparing and matching these multiple expression combinations with preset expressions, and what proportion of expressions match the preset expressions. To achieve a matching expression, you can gradually test the appropriate preset threshold range by setting values at different granularities, or you can reversely judge whether the threshold setting is appropriate through the recognition effect.
  • multimedia data is rendered based on text feature data, sentiment index and topic feature data.
  • the speech text obtained by speech conversion in the text feature data audio stream has an emotional index of the interviewer's emotional index.
  • the topic feature data is compared and matched based on the speech text and the preset text.
  • the matched preset text is Mapping topic feature data, or topic feature data obtained after matching the preset text through multi-level mapping and executing actions. Therefore, the multimedia data is rendered based on the text feature data, emotion index and topic feature data, and the text feature data is Render into the video and become the subtitles of the conference conversation. Render the emotional index into the video to display the emotional state of the participants in real time.
  • the text feature data obtained from the audio stream will
  • the topic feature data of the secondary processing content obtained through mapping or multi-level mapping of the text feature data, and the emotional index obtained from the video stream are rendered with the multimedia data respectively, so that the video stream not only contains the picture content of the meeting, but also includes Multi-dimensional auxiliary information is returned to the media plane routing, and sent to the conference site through the transmission system.
  • the multimedia data is parsed and presented through XR equipment at the conference site. According to 3Gpp specifications, this technical solution can be deployed on the routing or media processing network elements, which can also be deployed independently. Participants can see the original video information and hear the original audio information.
  • the meeting discusses the topic of financial security, a number of financial-related short sentences are scattered from the text feature data.
  • the preset financial-related short sentences For example, the stock market, names of listed companies, financial reports of listed companies, insurance companies, China Securities Regulatory Commission, etc., match a number of financial-related short sentences dispersed in the text feature data with the preset financial short sentences.
  • the preset threshold For example, when 10 short sentences or keywords are reached within a certain period of time, it is considered to have reached the preset threshold, that is, the successfully matched short sentences or keywords are mapped.
  • the mapping content can be the corresponding topic keywords, or it can be The content obtained after multi-level actions.
  • the successfully matched keywords are the names and financial data of listed companies.
  • the preset short sentences map to the actions.
  • the action content is "retrieve its income data in the past 2 years.” Therefore, based on matching If the keywords listed company name and financial data are found, the mapping content is to retrieve the revenue data of the corresponding listed company in the past two years. At the meeting site, the revenue data of the corresponding listed company in the past two years will be immediately displayed in the space, allowing participants to Participants can grasp the topic content more efficiently.
  • mapping relationships for various scenarios and topics can be preset.
  • the mapping relationships can also be multi-level mappings.
  • mapping relationships can be keywords and can also include execution actions. , also contains multi-level actions. After a lot of training and accumulation, the mapping relationship can be continuously optimized to become more accurate and intelligent. The accuracy and intelligence of the mapping relationship can also be judged by the length of time the participants visually pay attention to the content on site. , to achieve automated optimization of mapping relationships and automatic entry and update of mapping relationships.
  • mapping relationship includes at least one of: a mapping relationship between a preset micro-expression combination and an emotional index; a mapping relationship between a preset key phrase and a scene template; a mapping relationship between a preset statement statement and a command sequence; a preset sensitivity
  • mapping relationship between statements and rendering sets Setting up mapping relationships in multiple dimensions enables data processing to be performed based on the information in different dimensions and using mapping relationships in different dimensions.
  • step S200 may include but is not limited to the following sub-steps:
  • Step S2001 Perform frequency feature learning on the audio stream to obtain the frequency feature band
  • Step S2002 filter the audio stream using frequency characteristic bands to obtain several characteristic audio streams
  • Step S2003 Perform speech recognition on the characteristic audio stream to obtain subtitle text
  • Step S2004 Perform sound intensity analysis on the characteristic audio stream to obtain a sound intensity value.
  • the sound intensity value reaches a preset threshold, additional subtitle text is output.
  • frequency feature learning is performed on the audio stream, that is, the frequency features of different voices are learned, and the frequency feature band is obtained, that is, the frequency band where the frequency features of different voices are located.
  • the audio stream is Filter to obtain several characteristic audio streams, that is, separate the human voice from the audio stream, and further separate the voices of different people in the human voice to obtain the audio information of several channels of human voices.
  • Perform speech recognition on the characteristic audio stream to obtain For several subtitle texts corresponding to human voices, perform sound intensity analysis on the audio stream to obtain the sound intensity value.
  • additional subtitle text corresponding to several human voices is output, that is, short sentences whose pitch or volume exceeds the preset value. or keywords.
  • step S300 may include but is not limited to the following sub-steps:
  • Step S3001 Input the video stream to the preset deep learning model to identify and obtain facial area coordinates
  • Step S3002 Based on the coordinates of the facial area, perform micro-expression recognition and segmentation on the continuous frames of the facial area in the video stream to obtain a micro-expression combination.
  • facial expressions involve movements of multiple parts, among which "micro-expressions" naturally correspond to "macro-expressions". Macro expressions are easy to observe in our daily life. The duration of facial expressions is within 0.5-4 seconds, and the facial muscles involved in facial expressions contract or relax to a large extent. Micro-expressions are not only not controlled by thinking consciousness, but also last for a very short time. People's consciousness has been exposed before they have time to control them, so they reflect true emotions to a certain extent. It is precisely because of this characteristic of micro-expressions that it has very important applications in criminal investigation, security, justice, negotiation and other fields. Microexpression is a special facial expression. Compared with ordinary expressions, microexpressions mainly have the following characteristics:
  • the duration is short, usually only 1/25s ⁇ 1/3s;
  • the action intensity is low and difficult to detect
  • micro-expressions usually needs to be done in videos, while ordinary expressions can be analyzed in images.
  • micro-expressions The recognition of micro-expressions is initially trained manually. After about an hour and a half of training, the accuracy can be improved to 30%-40%, but psychologists have demonstrated that manual recognition will not exceed 47%. Later, as psychological experiments gradually evolved into computer applications, micro-expressions had AU (Action Units) combinations. For example, if it is happy, the happy macro-expression is AU6+AU12, and the micro-expression is AU6 or AU12, or AU6 +AU12. The second is micro-expression movement, which is a local muscle movement, rather than two muscle groups moving at the same time. For example, if you are happy, the movement is AU6 or AU12. But if the intensity is relatively high, AU6 and AU12 may occur at the same time.
  • AU Application Units
  • micro expressions can be extracted from videos through the micro expression recognition technology MER (Micro expression recognition).
  • MER Micro expression recognition
  • the expression changes of the participants can be accurately described, and the emotional changes of the participants can be reflected laterally, and the interest of the participants in the topic of discussion can be further analyzed. Therefore, as shown in Figure 3C, the video stream is input to the preset deep learning model.
  • the convolutional neural network (Convolutional Neural Networks, CNN) model can be used to first perform learning and training to obtain a mature CNN Model, input the video stream into the CNN model, perform face recognition, obtain the facial area coordinates through recognition, obtain the area coordinates of the faces of each interviewer in the video, use the MER blocker to perform MER block on the continuous frames of the video face area , further analyze the MER sequence, output the changes in the continuous time series pictures, and obtain the combination sequence of micro-expression AU, that is, the micro-expression combination.
  • CNN convolutional Neural Networks
  • the mapping relationship between the preset micro-expression combination and the emotional index is that the micro-expression combination consists of several micro-expressions, for example, micro-expression 1, micro-expression 2 and micro-expression 3, where micro-expression Expression 1 is AU6, micro-expression 2 is AU7, micro-expression 3 is AU12, that is, the micro-expression combination is AU6+AU7+AU12.
  • the micro-expressions can be arranged and combined to obtain a set of micro-expression combinations. According to the micro-expression set The corresponding expressed emotion is mapped to the corresponding emotion index.
  • the emotion index can be expressed in color, numerical, graphical and other forms. For example, set the emotion from angry to happy to 0-100 points, 0 means very angry, 100 means very happy. According to different expressions, set the corresponding emotion index.
  • the expressions extracted from the video stream can be matched to the corresponding scores.
  • the emotional index can be represented by color. Different colors correspond to different emotions, through color psychology, because warm colors represent boldness, sunshine, enthusiasm, and enthusiasm. , lively, and cold colors represent graceful, feminine, calm, and elegant.
  • step S200 also includes but is not limited to the following sub-steps:
  • Step S2005 process the subtitle text through natural language processing to obtain a sequence of short sentences
  • Step S2006 use the mapping relationship to compare and match the short sentence sequence with the preset key short sentences to obtain the corresponding scene template
  • Step S2007 Perform in-depth processing on the subtitle text and subtitle additional text according to the corresponding scene template to obtain topic feature data.
  • the subtitle text is semantically processed.
  • the subtitle text is semantically segmented into several short sentence sequences through natural language processing (Natural language processing), and a mapping relationship is used to combine the short sentence sequence with the predetermined sentence sequence.
  • Natural language processing Natural language processing
  • Set key phrases for comparison and matching When the matching degree reaches the preset threshold, the corresponding scene template is obtained.
  • the mapping relationship between preset key phrases and scene templates is that at least one preset key phrase is mapped to a scene.
  • a topic scene in the financial field is preset
  • the preset key phrase is the Shanghai Composite Index. , secondary market, Shanghai Stock Exchange, Shenzhen Stock Exchange, capital inflow and outflow in the two cities, A-shares, 6 preset key phrases mapped to financial scenarios
  • the subtitle text extracted from the video conference audio stream is divided into semantics Several short sentence sequences are compared and matched with 6 preset key short sentences, and the similarity is calculated. When the similarity reaches the preset threshold, it is judged that the current meeting topic is a financial topic, and then the financial scene is entered. .
  • the subtitle text and subtitle additional text are deeply processed to obtain topic feature data.
  • the subtitle text and the subtitle additional text are further semantically segmented according to the preset mapping relationship, and compared with the preset sentences to obtain action instructions for data processing.
  • the mapped Action process subtitle text and subtitle attached text for secondary processing, and output and render the secondary processing result.
  • the threshold can be set according to the strictness of scene matching and the number of preset key phrases. For example, in the same scene, the more the number of preset key phrases, the higher the matching degree required, then the threshold The higher it is, the more difficult it is to match the scene, but the accuracy will be higher.
  • this solution is preset with multiple scenarios, and multiple preset key phrases map to one scenario. Scenarios of different dimensions and levels can be set to apply to different types of meeting themes.
  • step S2007 also includes but is not limited to the following sub-steps:
  • Step S20071 use the mapping relationship to compare and match the short sentence sequence and subtitle additional text with the preset statement statement to obtain the corresponding command sequence;
  • Step S20072 executes the command sequence to obtain topic feature data.
  • mapping relationship that is, the mapping relationship between the preset declaration statement and the command sequence
  • the short sentence sequence and the subtitle additional text are compared and matched with the preset declaration statement to obtain the corresponding command sequence.
  • the mapping relationship between preset declaration statements and command sequences is that at least one preset declaration statement is mapped to a command sequence.
  • the command sequence contains an action group, that is, several data processing actions, for example, presetting a 5G communication field Topic scenarios, the preset key phrases are 5G networking, non-standalone networking NSA, independent networking SA, control bearer separation, mobile edge computing, network slicing, 6 preset key phrases are mapped to 5G communication scenarios, Segment the subtitle text extracted from the video conference audio stream into several short sentence sequences according to semantics, compare and match these short sentence sequences with 6 preset key short sentences, and calculate the similarity.
  • the mapping relationship between the preset declaration statement and the command sequence is used to attach text to the short sentence sequence and subtitles and the preset declaration.
  • the statements are compared and matched to obtain the corresponding command sequence.
  • the preset statement statements are the number of 5G patents, 5G standard essential patents, 5G patent holdings, 5G technology companies, 5G market size, and 5 preset statement statements.
  • the command sequence mapped by the five preset declaration statements can be set as:
  • Action 1 The search engine retrieves the preset declaration statement
  • Action 2 Extract the content introduction or keywords of the first five URLs in the search results
  • Action 3 Search the patent database for keywords extracted from the web page
  • Action 4 Remove duplicates from the search results and output them.
  • the command sequence contains 4 actions. Through this mapping relationship, several short sentence sequences are compared and matched with 5 preset statement sentences in 6 5G communication scenarios. When the similarity reaches the preset value, the match is considered successful. Therefore, the command sequence mapped by the preset declaration statement is executed, that is, 4 actions, and the action result is finally output.
  • the threshold can be set according to the strictness of the match and the number of preset declaration statements. For example, under the same command sequence, the more the number of preset declaration statements, the higher the matching degree required, the higher the threshold. , the more difficult it is to match the command sequence, but the accuracy will be higher. Therefore, a suitable threshold range can be found through a large amount of data training, taking into account both accuracy and practicality. It should be noted that multiple command sequences are preset in this solution. Multiple command sequences map to one command sequence. Command sequences of different dimensions and levels can be set to apply to different types of secondary data processing scenarios.
  • step S20071 also includes but is not limited to at least one of the following actions:
  • the command sequence includes an online search instruction
  • the online database is searched and the online search data set is obtained
  • the offline database is searched to obtain the offline retrieval data set
  • the command sequence contains data processing instructions
  • the data is processed twice to obtain secondary processing data.
  • search engine To search the database online, you can use one or more combinations of Google search engine, Baidu search engine, Bing search engine, Sogou search engine, Haosou search engine, Shenma search engine, and Yahoo search engine.
  • Search engines in subdivided professional fields such as special databases in the field of patent retrieval, special databases in the field of literature retrieval, etc.
  • offline databases are local databases, which can be internal databases in the local area network, targeting the fields used by this technical solution or the user's own
  • There is a database which can be set flexibly; when the command sequence contains data processing instructions, the data will be processed twice.
  • the secondary processing method can still use mapping relationships to perform multiple mappings and process the data multiple times and in multiple dimensions.
  • step S400 also includes but is not limited to the following sub-steps:
  • Step S4001 use the subtitle text to obtain the corresponding sound time sequence when the characteristic audio stream is started;
  • Step S4002 use the micro-expression combination to obtain the corresponding mouth shape time sequence when the mouth expression is activated;
  • Step S4003 determine the consistency of the sound time series and the mouth shape time series, and align the subtitle text, emotion index and facial area coordinates in time series;
  • Step S4004 use the mapping relationship to compare and match the short sentence sequence and the preset sensitive sentences to obtain the corresponding rendering set;
  • Step S4005 Render the subtitle text, emotion index, and topic feature data with the video stream according to the configuration of the rendering set.
  • the subtitle text is used to obtain the sound time sequence corresponding to the start of the characteristic audio stream.
  • the subtitle text is extracted with a time mark, and the time corresponding to the text can be determined; the micro-expression combination is used to obtain the corresponding sound time sequence when the mouth expression is started.
  • the micro-expression combination When extracting micro-expressions from the video stream, they are extracted sequentially according to the time series, so the micro-expression combination also has a time stamp.
  • the video stream and audio stream are the same multimedia source, they have consistent timing, so , during data processing, due to the separate processing of audio streams and video streams and secondary data processing, data timing may occur.
  • Asynchronous therefore, determine the consistency of the sound time series and the mouth shape time series, and align the subtitle text, emotion index and facial area coordinates in time series.
  • the subtitle text corresponds to the sound time series
  • the emotion index corresponds to the mouth shape time.
  • sequence, the facial area coordinates correspond to the mouth shape time series.
  • the short sentence sequence and the preset sensitive statement are compared and matched using the mapping relationship to obtain the corresponding rendering set.
  • the preset sensitive statement and the rendering set setting have a mapping relationship.
  • At least one preset sensitive statement and the rendering set are Mapping, according to the needs of the usage scenario, preset sensitive statements of different dimensions can be preset as matching references.
  • the configuration of the rendering set includes at least one of rendering coordinates, rendering time, and rendering color theme.
  • Different rendering sets will present different visual effects when rendering subtitle text, emotion index, and topic feature data with the video stream.
  • the rendering set after the subtitle text, emotion index and topic feature data are rendered with the video stream, meeting participants will see different colors of subtitle text and different styles near different faces. Sentiment index, topic feature data at different display times. Participants can see the nearby areas of different interviewers on the XR device. There are display boards or barrage pop-ups with subtitles for their respective talks. They can also see the display of the emotional index of the participants. The key information during the meeting will be processed after secondary processing.
  • the display duration, font color, and display position of different levels of key information are different.
  • the presentation method can be icons or numbers, or other designs.
  • different key information can be displayed through the management interface. Words and sentences are grouped, and the grouped collection can be set with corresponding processing actions and different rendering effect settings.
  • the embodiment of the present application also provides a multimedia data processing device, which can parse, extract, intelligently analyze and render the multimedia information of multi-person discussion calls, thereby improving the voice and video quality of existing call products.
  • multi-dimensional value information is mined, including voice text and emotional index, and the content of the voice text is further processed according to the preset mapping relationship, so that the call product can present richer and smarter information.
  • it can flexibly handle different business scenarios and meeting contents, expand the application areas of technical solutions, and effectively tap the potential value of call products, bringing a new experience to call discussions and increasing the number of negotiation business users. Improve negotiation competitiveness, improve decision-making efficiency, win more voice, and effectively seize business opportunities.
  • the multimedia data processing device includes:
  • the audio processing module 501 is configured to receive audio data, analyze and extract sounds in the audio stream, and obtain text feature data;
  • the video processing module 502 is configured to receive video data, analyze the video stream, and obtain expression feature data.
  • the expression feature data includes the expressions of the participants in the video session, and analyzes the people in the video through the recognition of consecutive frames in the video stream. face, extract the dynamic changes in facial expressions, and generate different expression feature data for different expressions;
  • the mapping relationship module 503 converts the text feature data into topic feature data according to the preset mapping relationship; converts the expression feature data into emotional index;
  • the rendering module 504 renders and outputs text feature data, sentiment index and topic feature data.
  • the audio processing module receives the audio data, analyzes and extracts the sounds in the audio stream, and obtains text feature data;
  • the video processing module receives the video data, analyzes the video stream, and obtains the expression feature data.
  • the expression feature data includes The expressions of the participants in the video conversation are analyzed through the recognition of consecutive frames in the video stream, and the dynamic changes in facial expressions are extracted to generate different expression feature data for different expressions; mapping relationship The module converts text feature data into topic feature data and expression feature data into emotion index according to the preset mapping relationship; the rendering module renders and outputs text feature data, emotion index and topic feature data. Based on this, this application is approved Analyze, extract, intelligently analyze and render the multimedia information of multi-person discussion calls.
  • an embodiment of the present application also provides an electronic device.
  • multi-dimensional value information can be mined, including voice text, emotional index, and The content of the voice text is further processed according to the preset mapping relationship, so that the call product can present richer and smarter information.
  • the application fields of the solution can effectively tap the potential value of call products, bring a new experience to call negotiations, increase the negotiation competitiveness of negotiation business users, improve decision-making efficiency, win more voice, and effectively seize business opportunities. .
  • the electronic equipment includes:
  • the processor can be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is configured to execute Relevant procedures to implement the technical solutions provided by the embodiments of this application;
  • a general-purpose CPU Central Processing Unit, central processing unit
  • a microprocessor central processing unit
  • an application specific integrated circuit Application Specific Integrated Circuit, ASIC
  • ASIC Application Specific Integrated Circuit
  • Memory can be implemented in the form of read-only memory (Read Only Memory, ROM), static storage device, dynamic storage device, or random access memory (Random Access Memory, RAM).
  • ROM read-only memory
  • RAM random access memory
  • the memory can store operating systems and other application programs.
  • the relevant program codes are stored in the memory and called by the processor to execute the goals of the embodiments of this application. How to sort data;
  • Input/output interface configured to implement information input and output
  • the communication interface is configured to realize communication interaction between this device and other devices. Communication can be achieved through wired methods (such as USB, network cables, etc.) or wireless methods (such as mobile network, WIFI, Bluetooth, etc.);
  • Buses which transmit information between various components of a device (such as processors, memory, input/output interfaces, and communication interfaces);
  • the processor, memory, input/output interface and communication interface realize the communication connection between each other within the device through the bus.
  • the electronic device includes: one or more processors and memories.
  • one processor and memory are taken as an example.
  • the processor and memory can be connected through a bus or other means.
  • Figure 7 takes the connection through a bus as an example.
  • the memory can be used to store non-transitory software programs and non-transitory computer executable programs, such as the multimedia data processing method in the above embodiments of the present application.
  • the processor implements the multimedia data processing method in the above embodiments of the present application by running non-transient software programs and programs stored in the memory.
  • the memory may include a program storage area and a data storage area, where the program storage area may store an operating system and an application program required for at least one function; the storage data area may store information required to execute the multimedia data processing method in the embodiments of the present application. Data etc.
  • the memory may include high-speed random access memory and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device.
  • the memory includes memory located remotely relative to the processor, and these remote memories may be connected to the multimedia media via a network. Data processing device. Examples of the above-mentioned networks include but are not limited to the Internet, intranets, local area networks, mobile communication networks and combinations thereof.
  • the non-transitory software programs and programs required to implement the multimedia data processing method in the above embodiments of the present application are stored in the memory.
  • the multimedia data processing method in the above embodiments of the present application is executed. , for example, perform the above-described method steps S100 to step S400 in Figure 1, method steps S2001 to step S2004 in Figure 2A, method steps S3001 to step S3002 in Figure 3A, and method steps S2005 to step S2007 in Figure 4A. .
  • this application analyzes and extracts, intelligently analyzes and renders the multimedia information of multi-person discussion calls, and digs out multi-dimensional value information based on the voice and video of existing call products, including voice text, emotion Index, and the content of the voice text is further processed according to the preset mapping relationship, so that the call product can present richer and smarter information.
  • mapping relationships can flexibly handle different business scenarios and talks.
  • Content expand the application fields of technical solutions, can effectively tap the potential value of call products, bring a new experience to call negotiations, increase the negotiation competitiveness of negotiation business users, improve decision-making efficiency, and win more voice. Seize business opportunities effectively.
  • embodiments of the present application also provide a computer-readable storage medium that stores a computer-executable program, and the computer-executable program is executed by one or more control processors, for example, as shown in FIG. 7
  • Execution by one of the processors may cause the one or more processors to execute the multimedia data processing method in the embodiment of the present application, for example, execute the above-described method steps S100 to S400 in FIG. 1, and execute steps S400 in FIG. 2A.
  • this application analyzes and extracts, intelligently analyzes and renders the multimedia information of multi-person discussion calls, and digs out multi-dimensional value information based on the voice and video of existing call products, including voice text, emotion Index, and the content of the voice text is further processed according to the preset mapping relationship, so that the call product can present richer and smarter information.
  • mapping relationships can flexibly handle different business scenarios and talks.
  • Content expand the application fields of technical solutions, can effectively tap the potential value of call products, bring a new experience to call negotiations, increase the negotiation competitiveness of negotiation business users, improve decision-making efficiency, and win more voice. Seize business opportunities effectively.
  • Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by a computer.
  • communication media typically embodies a computer-readable program, data structure, program module or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Acoustics & Sound (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Biology (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)

Abstract

本申请公开了多媒体数据处理方法、装置和计算机可读存储介质。该方法包括，获取多媒体数据的音频流和视频流（S100）；解析音频流，得到文本特征数据，根据预设的映射关系对照匹配文本特征数据以确定话题特征数据（S200）；解析视频流，得到表情特征数据，根据预设的映射关系对照匹配表情特征数据以确定情感指数（S300）；基于文本特征数据、情感指数和话题特征数据对多媒体数据进行渲染（S400）。

Description

多媒体数据处理方法、装置和计算机可读存储介质
相关申请的交叉引用
本申请基于申请号为202210722924.7、申请日为2022年6月24日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此引入本申请作为参考。
技术领域
本申请实施例涉及但不限于数据处理技术领域，特别是涉及一种多媒体数据处理方法、装置和计算机可读取存储介质。
背景技术
随着经济全球化，跨地域的商务沟通越发频繁，目前的远程商务会议多通过视频会议系统进行，通过传输线路及多媒体设备，将声音、影像及文件资料互传，实现即时且互动的沟通，随着增强现实技术的发展以及5G技术高通量低延时确定性网络的出现，为视频通话乃至增强现实场景带来了高性能的处理潜力，同时，业务层面也需要商务视频会议能够从数据通道中挖掘提供更多有效的会谈辅助信息，然而，当前主流的商务会议通话产品仅仅提供视频语音通话控制和媒体通道服务，缺乏对通道数据进行多维度的分析挖掘和二次加工，没有对媒体数据进行有效的挖掘利用。
发明内容
以下是对本文详细描述的主题的概述。本概述并非是为了限制权利要求的保护范围。
本申请实施例提供了一种多媒体数据处理方法、装置和计算机可读存储介质。
第一方面，本申请实施例提供了一种多媒体数据处理方法，包括：获取多媒体数据的音频流和视频流；解析所述音频流，得到文本特征数据，根据预设的映射关系对照匹配所述文本特征数据以确定话题特征数据；解析所述视频流，得到表情特征数据，根据预设的映射关系对照匹配所述表情特征数据以确定情感指数；基于所述文本特征数据、所述情感指数和所述话题特征数据对所述多媒体数据进行渲染。
第二方面，本申请实施例提供了一种多媒体数据处理装置，包括：音频处理模块，被配置为接收并解析音频流，获得文本特征数据；视频处理模块，被配置为接收并解析视频流，得到表情特征数据；映射关系模块，被配置为处理所述文本特征数据得到话题特征数据，处理所述表情特征数据得到情感指数；渲染模块，被配置为将所述文本特征数据、所述话题特征数据和所述情感指数与所述视频流渲染。
第三方面，本申请实施例提供了一种电子设备，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如上第一方面所述的多媒体数据处理方法。
第四方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存 储有计算机可执行程序，所述计算机可执行程序用于使计算机执行如上第一方面所述的多媒体数据处理方法。
本申请的其它特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请而了解。本申请的目的和其他优点可通过在说明书、权利要求书以及附图中所特别指出的结构来实现和获得。
附图说明
附图用来提供对本申请技术方案的进一步理解，并且构成说明书的一部分，与本申请的实施例一起用于解释本申请的技术方案，并不构成对本申请技术方案的限制。
图1是本申请一个实施例提供的一种多媒体数据处理方法的主流程图；
图2A是本申请一个实施例提供的一种多媒体数据处理方法的子流程图；
图2B是本申请一个实施例提供的解析音频流得到文本特征数据的子流程图；
图3A是本申请一个实施例提供的一种多媒体数据处理方法的子流程图；
图3B是本申请一个实施例提供的微表情的示意图；
图3C是本申请一个实施例提供的解析视频流得到表情特征数据的子流程图；
图3D是本申请一个实施例提供的预置微表情组合与情感指数的映射关系的示意图；
图4A是本申请一个实施例提供的一种多媒体数据处理方法的子流程图；
图4B是本申请一个实施例提供的预置关键短句与场景模板的映射关系的示意图；
图4C是本申请一个实施例提供的一种多媒体数据处理方法的子流程图；
图4D是本申请一个实施例提供的预置声明语句与命令序列的映射关系的示意图；
图4E是本申请一个实施例提供的命令序列类型的示意图；
图5A是本申请一个实施例提供的一种多媒体数据处理方法的子流程图；
图5B是本申请一个实施例提供的预置敏感语句与渲染集的映射关系的示意图；
图6是本申请一个实施例提供的一种多媒体数据处理装置结构的示意图；
图7是本申请一个实施例提供的电子设备结构示意图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
应了解，在本申请实施例的描述中，多个(或多项)的含义是两个以上，大于、小于、超过等理解为不包括本数，以上、以下、以内等理解为包括本数。如果有描述到“第一”、“第二”等只是用于区分技术特征为目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量或者隐含指明所指示的技术特征的先后关系。
当今经济全球化，跨地域的商务沟通越发频繁，目前的远程商务会议多通过视频会议系统进行，通过传输线路及多媒体设备，将声音、影像及文件资料互传，实现即时且互动的沟通，随着XR技术的发展以及5G技术高通量低延时确定性网络的出现，为视频通话乃至XR场景带来了高性能的处理潜力，AR、VR、XR技术的发展使得远程通话技术越来越接近出差现场谈话的效果，虽然XR设备的崛起，使得原来一维的语音通话到二维的视频通话，增加了三维 空间通话，但是，媒体数据还没有有效的挖掘利用出来，当前的通话主流产品仅仅提供了视频语音通话控制和媒体通道服务，而对通道数据进行多维度分析的产品少之又少，业务层面也需要商务视频会议能够从数据通道中挖掘提供更多有效的会谈辅助信息。
同时，通讯数据通道和数据处理能力的不断发展，通讯运营商提供能力开放平台，引入第三方开发者来提供丰富多彩的业务的想法由来已久，但是在实践中，国内运营商部署的能力开放平台依然没有取得预定的效果，从根本上来讲，这些开放的能力，还是只能针对原有的语音和视频呼叫做一些简单的控制，从用户体验上来说没有产生突破性的改变。
针对现有视频通话产品的不足之处和业务需要，本申请实施例提供了一种多媒体数据处理方法、装置和计算机可读存储介质，首先，获取多媒体数据的音频流和视频流；解析音频流，得到文本特征数据，根据预设的映射关系对照匹配文本特征数据以确定话题特征数据；解析视频流，得到表情特征数据，根据预设的映射关系对照匹配表情特征数据以确定情感指数；基于文本特征数据、情感指数和话题特征数据对多媒体数据进行渲染。基于此，本申请通过对多人商谈通话的多媒体信息进行解析提取、智能分析和渲染增强，在现有通话产品的语音和视频的基础上，挖掘出多维度的价值信息，包括语音文本、情感指数，并且将语音文本的内容根据预设的映射关系进一步二次加工，使得通话产品能够呈现更丰富智能的信息，同时，通过预设不同的映射关系，能够灵活的处理不同的商务场景和会谈内容，扩展技术方案的应用领域，可以有效的挖掘通话类产品的潜在价值，给通话商谈带来全新的体验，增加商谈业务使用者的会谈竞争力，提高决策效率，赢得更多的话语权，有效抓住商机。
如图1所示，图1是本申请一个实施例提供的一种多媒体数据处理方法的流程图。多媒体数据处理方法包括但不限于如下步骤：
步骤S100，获取多媒体数据的音频流和视频流；
步骤S200，解析音频流，得到文本特征数据，根据预设的映射关系对照匹配文本特征数据以确定话题特征数据；
步骤S300，解析视频流，得到表情特征数据，根据预设的映射关系对照匹配表情特征数据以确定情感指数；
步骤S400，基于文本特征数据、情感指数和话题特征数据对多媒体数据进行渲染。
可以理解的是，远程商务会议采用视频通话的方式，将会议两端的现场多媒体信息交换传输给对方，网络中传输的多媒体包括音频和视频，在本技术方案中，首先获取多媒体数据的音频流和视频流，将音频流中的声音进行解析提取，获得文本特征数据，例如，音频流中的声音可以是人声，将人声通过语音识别技术转换成文本，即语言文字；类似的，也可以是非人声，根据频谱，识别出其对应的声音类型、名称、特点等，并将声音对应的名称或者动作转换成文本的描述，例如，一种乐器的声音，识别到是钢琴曲，文本特征数据可以包括钢琴曲、钢琴曲的名称、钢琴曲的相关信息等；然后将多媒体数据中的视频流进行解析，得到表情特征数据，表情特征数据包括视频会话中参与人的表情，例如，通过视频流中连续帧的识别，分析出视频中的人脸，并将人脸表情的动态变化提取出来，针对不同的表情，生成不同的表情特征数据，例如，识别出人脸的愤怒表情，通过眉毛，眼睛，脸庞，鼻子，嘴唇等姿态的组合，生成表情特征数据。
可以理解的是，根据预设的映射关系对照匹配文本特征数据以确定话题特征数据，预设的映射关系是将一些文本特征数据跟话题特征数据设置为映射关系，即将一些文本短句跟话 题关键词设置为映射关系，当然，也可以将文本短句跟一些动作命令设置为映射关系，例如，将文本短句匹配为搜索动作，即如果达成匹配，则对匹配到的短句进行检索，将搜索结果作为话题特征数据，当然也可以即将映射关系预设为多级映射关系，即将预置的文本短句跟话题关键词设置为映射关系，进一步将话题关键词跟动作命令设置为映射关系，还可以将动作命令继续跟另一动作命令设置为映射关系，自此，经过多级的映射关系设置，即文本特征数据经过多级映射转化和动作处理后得到话题特征数据，因此，根据预设的映射关系，将文本特征数据中的文本短句跟映射关系中预置的文本短句进行匹配比对，若匹配度达到预设阈值，根据预设的映射关系，将文本特征数据中的文本短句匹配到的预置文本短句所映射的关键词确定为话题特征数据。在一些实施例中，文本特征数据可以包含一整篇文本，在进行文本匹配时，将文本特征数据里的文本分散为短句或者短词，分别将分散的短句或者短词跟预置的文本短句进行匹配，预设阈值可以理解为文本特征数据里分散的短句或者短词文本短句有多少比例跟预置的文本短句达成匹配，可以通过设置不同颗粒度的数值，逐步测试合适的预设阈值范围，也可以通过识别的效果判断阈值设置是否合适。根据预设的映射关系对照匹配表情特征数据以确定情感指数，同样的，预设的映射关系是将一些表情特征数据跟情感指数设置为映射关系，即将视频流中提取出来的表情跟情感指数设置为映射关系，情感指数可以根据颜色、数值、图形化等形式表示，例如，将情绪从愤怒到开心设置为0-100分，0分为非常愤怒，100分为非常开心，根据不同的表情，设置对应的情感指数，即可将视频流中提取出来的表情匹配到对应的分数上，同样的，可以通过颜色表示情感指数，不同的颜色对应不同的情绪，通过色彩心理学，因为暖色代表豪放、阳光、热情、热烈、活泼，冷色代表婉约、阴柔、冷静、文雅，因此，可以设置红色代表热烈、轰轰烈烈、冲动；橙色代表快乐、幸福；黄色代表骄傲；绿色代表平和；蓝色代表冷漠、平静、理智、冷酷；紫色代表梦幻；黑色和白色代表恐怖、绝望、悲哀、崇高；灰色代表冷静，将不同的情绪表情根据色彩心理学映射为不同的颜色，根据视频流中提取出来的表情，将之跟预置的表情对照匹配，若匹配度达到预设阈值，根据预设的映射关系，将表情特征数据中的表情匹配到的预置表情所映射的颜色确定为情感指数；但同样，使用图形化也是类似的方式，用图形的圆满度来表示情绪的变化，在此不再做举例。表情特征数据可以是包含一段时间内的表情变化，可能包括多个表情组合，预设阈值可以理解为将此多个表情组合跟预置的表情做比对匹配，有多少比例的表情跟预置的表情达成匹配，可以通过设置不同颗粒度的数值，逐步测试合适的预设阈值范围，也可以通过识别的效果反向判断阈值设置是否合适。
可以理解的是，基于文本特征数据、情感指数和话题特征数据对多媒体数据进行渲染。文本特征数据音频流中的语音转化得到的语音文本，情感指数为会谈人的情绪指数，话题特征数据是根据语音文本与预置文本进行比对匹配，根据映射关系，将匹配的预置文本所映射的话题特征数据，或者将匹配的预置文本经过多级映射并执行动作之后得到的话题特征数据，因此，基于文本特征数据、情感指数和话题特征数据对多媒体数据进行渲染，将文本特征数据渲染到视频中，成为会议交谈的字幕，将情感指数渲染到视频中，实时展现参会人员的情绪状态，便于及时根据参会人员的情绪调整话题或者会议内容，将话题特征数据渲染到视频中，是对会议交谈关键信息的二次加工，便于提高会议交谈的效率，或者因为参会人员的知识水平参差不齐或者信息不对称的原因，通过关键信息的二次加工和渲染，及时显示相关话题的相关内容，使参会人员及时把握会议内容，例如，将从音频流得到的文本特征数据，将 文本特征数据经过映射或者多级映射得到的二次加工内容的话题特征数据，和视频流中得到的情感指数，分别跟多媒体数据进行渲染，使得视频流中不只包含会议的画面内容，还包括了多维度的辅助信息，将此视频流返回给媒体面路由，经过传输系统送到会议现场，在会议现场通过XR设备，将多媒体数据进行解析和呈现，按照3Gpp规范，本技术方案可以部署在路由或媒体处理网元，也可以独立部署，参会人员可以看到原有视频信息，可以听到原有的音频信息，同时，可以看到相关参会人的语音字幕，看到参会人的情感指数，可以即时看到相关话题的总结或者相关信息呈现，当会议在讨论金融安全的话题时，从文本特征数据中分散出若干包含金融相关的短句，根据预设的金融相关短句，例如股市，上市公司名称，上市公司财报，保险公司，证监会等，将文本特征数据中分散出的若干包含金融相关的短句与预设的金融短句进行匹配，当到达预设阈值时，例如一定时间内达到10个短句或者关键词的时，即认为达到了预设的阈值，即对匹配成功的短句或者关键词进行映射，映射内容可以是对应的话题关键词，也可以是经过多级动作得到的内容，例如匹配成功的关键词是上市公司名称和财务数据，预置短句映射的是动作，同时动作内容为“检索其近2年的收入数据”，因此，基于匹配到的关键词上市公司名称和财务数据，则映射的内容为检索对应上市公司近2年的收入数据，在会议现场，立刻会在空间中展现出对应上市公司近2年的收入数据，使参会人员更高效的把握话题内容，需要说明的是，可以预设针对各种场景和话题的映射关系，映射关系也可以是多级映射，同时，映射关系可以是关键词，也可以包含执行动作，同样包含多级动作，映射关系经过大量的训练和积累可以不断优化，达到更准确和智能，也可以通过参会人员现场视觉关注内容停留的时间长短，判断映射关系的关联准确度和智能性，实现自动化的优化映射关系和自动录入更新映射关系。
可以理解的是，映射关系包括至少之一：预置微表情组合与情感指数的映射关系；预置关键短句与场景模板的映射关系；预置声明语句与命令序列的映射关系；预置敏感语句与渲染集的映射关系。设置多种维度的映射关系，可以使信息处理时，依照不同维度的信息，采用不同维度的映射关系进行数据处理。
如图2A所示，步骤S200可以包括但不限于如下子步骤：
步骤S2001，对音频流进行频率特征学习，得到频率特征频段；
步骤S2002，利用频率特征频段对音频流过滤，得到若干特征音频流；
步骤S2003，对特征音频流进行语音识别，得到字幕文本；
步骤S2004，对特征音频流进行音强分析，得到音强值，当音强值达到预设阈值，则输出字幕附加文本。
可以理解的，如图2B所示，对音频流进行频率特征学习，即学习不同人声的频率特征，得到频率特征频段，即不同人声的频率特征所在的频段，通过特征数据，对音频流进行过滤，得到若干特征音频流，即从音频流中将人声分离出来，并且进一步将人声中不同人的声音分离开，得到若干路人声的音频信息，对特征音频流进行语音识别，得到若干人声对应的字幕文本，对音频流进行音强分析，得到音强值。在一些实施例中，通过分析人声音频的音调高度或者声音大小，当音强值超过预设阈值时，则输出若干路人声对应的字幕附加文本，即音调或者音量超过预设值的短句或者关键词。
如图3A所示，步骤S300可以包括但不限于如下子步骤：
步骤S3001，将视频流输入到预置的深度学习模型，识别获得面部区域坐标；
步骤S3002，根据面部区域坐标，对视频流中面部区域的连续帧进行微表情识别分块，得到微表情组合。
如图3B所示，面部表情涉及多个部位的动作，其中“微表情”自然对应着的是“宏表情”。宏表情在我们平时比较容易观察到的，面部表情持续时间在0.5-4秒范围之内，且参与面部表情动作的面部肌肉群收缩或舒张幅度较大。微表情不但不受思维意识控制，而且持续的时间非常短，人的意识还没有来得及去控制它，就已经被暴露出来，所以它一定程度上反应了真实的情绪。正是由于微表情的这个特点，它在刑侦、安全、司法、谈判等领域都有非常重要的应用。微表情是一种特殊的面部表情，与普通的表情相比，微表情主要有以下特点：
1.持续时间短，通常只有1/25s～1/3s；
2.动作强度低，难以察觉；
3.在无意识状态下产生，通常难以掩饰或伪装；
4.对微表情的分析通常需要在视频中，而普通表情在图像中就可以分析。
微表情的识别，它最开始是通过人工训练的，大概训练一个半小时，可以提高到30％-40％的准确性，但心理学家论证了人工识别最高不会超过47％。后来随着心理学实验逐渐演进到计算机应用上面，微表情有了AU(Action Units)组合，例如，如果是高兴，宏表情的高兴是AU6+AU12，微表情是AU6或者是AU12，或者是AU6+AU12。第二是微表情运动，它是局部的肌肉块运动，而不是两个肌肉块同时运动，比如高兴，运动的是AU6或者是AU12。但如果强度比较大，可能会出现AU6和AU12同时发生的情况。另外宏表情和微表情有一些联系，就是微表情的AU可能是宏表情AU的子集，因此，通过微表情识别技术MER(Micro expression recognition)可以从视频中提取出微表情。通过微表情组合，即可准确描述参会人员的表情变化，侧面反应出参会人员的情绪变化，可以进一步分析出参会人员对讨论话题的兴趣程度。因此，如图3C所示，将视频流输入到预置的深度学习模型，在一些实施例中，可以采用卷积神经网络(Convolutional Neural Networks,CNN)模型，先进行学习训练，得到成熟的CNN模型，将视频流输入到CNN模型，进行人脸识别，识别获得面部区域坐标，获得视频中各个会谈人员面部的区域坐标，使用MER分块器，对视频人脸区域的连续帧进行MER分块，进一步MER序列解析，对连续时序图片的变化进行输出，得到微表情AU的组合序列，即微表情组合。
可以理解的是，如图3D所示，预置微表情组合与情感指数的映射关系，即微表情组合由若干微表情组成，例如，微表情1、微表情2和微表情3，其中，微表情1为AU6，微表情2为AU7，微表情3为AU12，即微表情组合为AU6+AU7+AU12，同样的，可以对微表情进行排列组合，得到微表情组合的集合，根据微表情集合所对应表达的情绪，映射为对应的情感指数，因此，通过对视频流中的参会人员的表情提取，得到微表情组合，并根据预置为表情组合的映射关系，得到情感指数，情感指数可以根据颜色、数值、图形化等形式表示，例如，将情绪从愤怒到开心设置为0-100分，0分为非常愤怒，100分为非常开心，根据不同的表情，设置对应的情感指数，即可将视频流中提取出来的表情匹配到对应的分数上，同样的，可以通过颜色表示情感指数，不同的颜色对应不同的情绪，通过色彩心理学，因为暖色代表豪放、阳光、热情、热烈、活泼，冷色代表婉约、阴柔、冷静、文雅，因此，可以设置红色代表热烈、轰轰烈烈、冲动；橙色代表快乐、幸福；黄色代表骄傲；绿色代表平和；蓝色代表冷漠、平静、理智、冷酷；紫色代表梦幻；黑色和白色代表恐怖、绝望、悲哀、崇高；灰色代表冷 静，将不同的微表情组合根据色彩心理学映射为不同的颜色，即可将微表情组合映射为情感指数。
如图4A所示，步骤S200还包括但不限于如下子步骤：
步骤S2005，将字幕文本通过自然语言处理，获得短句序列；
步骤S2006，利用映射关系，将短句序列与预置关键短句进行对照匹配，得到对应的场景模板；
步骤S2007，依据对应的场景模板，对字幕文本、字幕附加文本进行深层处理获得话题特征数据。
可以理解的是，将字幕文本进行语义处理，在一些实施例中，通过自然语言处理(Natural language processing)，将字幕文本按照语义分割成若干短句序列，利用映射关系，将短句序列与预置关键短句进行对照匹配，当匹配度达到预设阈值时，得到对应的场景模板。
如图4B所示，预置关键短句与场景模板的映射关系，至少一个预置关键短句与一个场景映射，例如，预置一个金融领域的话题场景，预置的关键短句为上证指数，二级市场，上海证券交易所，深圳证券交易所，两市资金流入流出，A股，6个预置关键短句与金融场景映射，将视频会议音频流提取出来的字幕文本按照语义分割成若干短句序列，将此若干短句序列跟6个预置关键短句进行比对匹配，计算相似度，当相似度达到预设阈值时，判断认为当前会议话题是金融话题，则进入金融场景。依据对应的场景模板，对字幕文本、字幕附加文本进行深层处理获得话题特征数据。在一些实施例中，在金融场景下，根据预置的映射关系，进一步将字幕文本和字幕附加文本进行语义分割，与预置的语句进行比对，得到对数据加工的动作指令，通过映射的动作，将字幕文本和字幕附加文本加工进行二次加工，将二次加工结果输出渲染。需要说明的是，阈值可以根据场景匹配的严格程度和预置关键短句的数量来设置，例如，同一个场景下，预置关键短句数量越多，要求达到的匹配度越高，则阈值越高，越难以与场景匹配，但是准确度会越高，因此，可以通过大量的数据训练，找到合适的阈值范围，兼顾准确度和实用性。需要说明的是，本方案中预置多种场景，多个预置关键短句映射一个场景，可以设置不同维度，不同级别的场景，来应用于不同类型的会议主题。
如图4C所示，步骤S2007还包括但不限于如下子步骤：
步骤S20071，利用映射关系，将短句序列和字幕附加文本与预置声明语句进行对照匹配，得到对应的命令序列；
步骤S20072执行命令序列，获得话题特征数据。
可以理解的是，利用映射关系，即预置声明语句与命令序列的映射关系，将短句序列和字幕附加文本与预置声明语句进行对照匹配，得到对应的命令序列.
如图4D所示，预置声明语句与命令序列的映射关系，至少一个预置声明语句与一个命令序列映射，命令序列包含一个动作组，即若干数据处理动作,例如，预置一个5G通信领域的话题场景，预置的关键短句为5G组网，非独立组网NSA，独立组网SA，控制承载分离，移动边缘计算，网络切片，6个预置关键短句与5G通信场景映射，将视频会议音频流提取出来的字幕文本按照语义分割成若干短句序列，将此若干短句序列跟6个预置关键短句进行比对匹配，计算相似度，当相似度达到预设阈值时，判断认为当前会议话题是5G通信话题，则进入5G通信场景，在5G通信场景内，对字幕文本、字幕附加文本做进一步的匹配映射。在一些实施例中，利用预置声明语句与命令序列的映射关系，将短句序列和字幕附加文本与预置声明 语句进行对照匹配，得到对应的命令序列，例如，预置声明语句为5G专利数量，5G标准必要专利，5G专利持有量，5G技术公司，5G市场规模，5个预置声明语句，同时，5个预置声明语句所映射的命令序列可以设置为：
动作1：搜索引擎检索预置声明语句；
动作2：提取检索结果前五条网址的内容简介或者关键词；
动作3：在专利数据库检索网页中提取的关键词；
动作4：将检索结果去重后输出。
命令序列包含4个动作，通过此映射关系，将若干短句序列与6个5G通信场景下的5个预置声明语句进行比对匹配，当相似度达到预设值时，即认为匹配成功，因此执行预置声明语句所映射的命令序列，即4个动作，最后输出动作结果。需要说明的是，阈值可以根据匹配的严格程度和预置声明语句的数量来设置，例如，同一个命令序列下，预置声明语句数量越多，要求达到的匹配度越高，则阈值越高，越难以与命令序列匹配，但是准确度会越高，因此，可以通过大量的数据训练，找到合适的阈值范围，兼顾准确度和实用性。需要说明的是，本方案中预置多种命令序列，多个命令序列映射一个命令序列，可以设置不同维度，不同级别的命令序列，来应用于不同类型的数据二级加工场景。
如图4E所示，步骤S20071还包括但不限于如下至少一种动作：
当命令序列包含在线检索指令，则检索在线数据库，得到在线检索数据集；
当命令序列包含离线检索指令，则检索离线数据库，得到离线检索数据集；
当命令序列包含数据加工指令，则对数据进行二次加工，得到二次加工数据。
在线检索数据库，可以采用谷歌搜索引擎、百度搜索引擎、必应搜索引擎、搜狗搜索引擎、好搜搜索引擎、神马搜索搜索引擎、雅虎搜索引擎中的一种或多种结合，当然也可以使用细分专业领域的搜索引擎，例如，专利检索领域的专用数据库，文献检索领域的专用数据库等；离线数据库是本地数据库，可以为局域网内的内部数据库，针对本技术方案所使用的领域或用户自有数据库，可以灵活设置；当命令序列包含数据加工指令，则对数据进行二次加工，二次加工的方法依然可以采用映射关系，进行多重映射，对数据进行多次多维度的处理。
如图5A所示，步骤S400还包括但不限于如下子步骤：
步骤S4001，利用字幕文本，获得特征音频流启动时对应的声音时间序列；
步骤S4002，利用微表情组合，获得嘴型表情启动时对应的嘴型时间序列；
步骤S4003，判断声音时间序列和嘴型时间序列的一致性，将字幕文本、情感指数和面部区域坐标在时序上对齐；
步骤S4004，利用映射关系，将短句序列和预置敏感语句进行对照匹配，得到对应的渲染集；
步骤S4005，根据渲染集的配置，将字幕文本、情感指数和话题特征数据与视频流渲染。
可以理解的是，利用字幕文本，获得特征音频流启动时对应的声音时间序列，字幕文本提取时带有时间标记，可以确定文本所对应的时间；利用微表情组合，获得嘴型表情启动时对应的嘴型时间序列，从视频流中提取微表情时，根据时间序列顺序提取，因此微表情组合也带有时间标记，同时，因为视频流和音频流是同一多媒体源，具有一致的时序，因此，在数据处理时，由于音频流和视频流分路处理，并且经过二次数据处理，可能会出现数据时序 不同步，因此，判断声音时间序列和嘴型时间序列的一致性，将字幕文本、情感指数和面部区域坐标在时序上对齐，字幕文本对应的是声音时间序列，情感指数对应的是嘴型时间序列，面部区域坐标对应的是嘴型时间序列，当判断声音时间序列和嘴型时间序列一致时，则字幕文本、情感指数和面部区域坐标在时序上也是对齐的。
如图5B所示，利用映射关系，将短句序列和预置敏感语句进行对照匹配，得到对应的渲染集，预置敏感语句与渲染集设置有映射关系，至少一句预置敏感语句与渲染集映射，根据使用场景的需要，可以预设不同维度的预置敏感语句作为匹配参照。
渲染集的配置包括渲染坐标、渲染时间、渲染颜色主题至少之一，不同的渲染集，在将字幕文本、情感指数和话题特征数据与视频流渲染时，会呈现出不同的视觉效果。例如，在XR会议中，根据渲染集的不同配置，字幕文本、情感指数和话题特征数据与视频流渲染后，会议参与人会在不同的人脸庞附近看到不同颜色的字幕文本，不同风格的情感指数，不同展示时间的话题特征数据。参会人员在XR设备中可以看到不同会谈人的附近区域，有显示板或者弹幕弹出各自的会谈字幕，看到参与人的情感指数的显示，会谈期间的关键信息经过二次加工后会反馈到视觉空间中同时，不同级别的关键信息显示时长、字体颜色、显示位置均表现不同，呈现的方式可以是图标或者数字，也可以是其他设计，同时，可以通过管理界面，对不同的关键词句进行分组，分组后的集合可以设置响应的处理加工动作，可以进行不同的渲染效果设置。
如图6所示，本申请实施例还提供了一种多媒体数据处理装置，可以通过对多人商谈通话的多媒体信息进行解析提取、智能分析和渲染增强，在现有通话产品的语音和视频的基础上，挖掘出多维度的价值信息，包括语音文本、情感指数，并且将语音文本的内容根据预设的映射关系进一步二次加工，使得通话产品能够呈现更丰富智能的信息，同时，通过预设不同的映射关系，能够灵活的处理不同的商务场景和会谈内容，扩展技术方案的应用领域，可以有效的挖掘通话类产品的潜在价值，给通话商谈带来全新的体验，增加商谈业务使用者的会谈竞争力，提高决策效率，赢得更多的话语权，有效抓住商机。
该多媒体数据处理装置包括：
音频处理模块501，被配置为接收音频数据，将音频流中的声音进行解析提取，获得文本特征数据；
视频处理模块502，被配置为接收视频数据，对视频流进行解析，得到表情特征数据，表情特征数据包括视频会话中参与人的表情，通过视频流中连续帧的识别，分析出视频中的人脸，并将人脸表情的动态变化提取出来，针对不同的表情，生成不同的表情特征数据；
映射关系模块503，根据预设的映射关系，将文本特征数据转换为话题特征数据；将表情特征数据转换为情感指数；
渲染模块504，将文本特征数据、情感指数和话题特征数据渲染输出。
在一实施方式中，音频处理模块接收音频数据，将音频流中的声音进行解析提取，获得文本特征数据；视频处理模块接收视频数据，对视频流进行解析，得到表情特征数据，表情特征数据包括视频会话中参与人的表情，通过视频流中连续帧的识别，分析出视频中的人脸，并将人脸表情的动态变化提取出来，针对不同的表情，生成不同的表情特征数据；映射关系模根据预设的映射关系，将文本特征数据转换为话题特征数据，将表情特征数据转换为情感指数；渲染模块将文本特征数据、情感指数和话题特征数据渲染输出。基于此，本申请通过 对多人商谈通话的多媒体信息进行解析提取、智能分析和渲染增强，在现有通话产品的语音和视频的基础上，挖掘出多维度的价值信息，包括语音文本、情感指数，并且将语音文本的内容根据预设的映射关系进一步二次加工，使得通话产品能够呈现更丰富智能的信息，同时，通过预设不同的映射关系，能够灵活的处理不同的商务场景和会谈内容，扩展技术方案的应用领域，可以有效的挖掘通话类产品的潜在价值，给通话商谈带来全新的体验，增加商谈业务使用者的会谈竞争力，提高决策效率，赢得更多的话语权，有效抓住商机。
如图7所示，本申请实施例还提供了一种电子设备。可以通过对多人商谈通话的多媒体信息进行解析提取、智能分析和渲染增强，在现有通话产品的语音和视频的基础上，挖掘出多维度的价值信息，包括语音文本、情感指数，并且将语音文本的内容根据预设的映射关系进一步二次加工，使得通话产品能够呈现更丰富智能的信息，同时，通过预设不同的映射关系，能够灵活的处理不同的商务场景和会谈内容，扩展技术方案的应用领域，可以有效的挖掘通话类产品的潜在价值，给通话商谈带来全新的体验，增加商谈业务使用者的会谈竞争力，提高决策效率，赢得更多的话语权，有效抓住商机。
该电子设备包括：
处理器，可以采用通用的CPU(Central Processing Unit，中央处理器)、微处理器、应用专用集成电路(Application Specific Integrated Circuit，ASIC)、或者一个或多个集成电路等方式实现，被配置为执行相关程序，以实现本申请实施例所提供的技术方案；
存储器，可以采用只读存储器(Read Only Memory，ROM)、静态存储设备、动态存储设备或者随机存取存储器(Random Access Memory，RAM)等形式实现。存储器可以存储操作系统和其他应用程序，在通过软件或者固件来实现本说明书实施例所提供的技术方案时，相关的程序代码保存在存储器中，并由处理器来调用执行本申请实施例的目标数据的排序方法；
输入/输出接口，被配置为实现信息输入及输出；
通信接口，被配置为实现本设备与其他设备的通信交互，可以通过有线方式(例如USB、网线等)实现通信，也可以通过无线方式(例如移动网络、WIFI、蓝牙等)实现通信；
总线，在设备的各个组件(例如处理器、存储器、输入/输出接口和通信接口)之间传输信息；
其中处理器、存储器、输入/输出接口和通信接口通过总线实现彼此之间在设备内部的通信连接。
在一实施方式中，该电子设备包括：一个或多个处理器和存储器，图7中以一个处理器及存储器为例。处理器和存储器可以通过总线或者其他方式连接，图7中以通过总线连接为例。
存储器作为一种非暂态计算机可读存储介质，可用于存储非暂态软件程序以及非暂态性计算机可执行程序，如上述本申请实施例中的多媒体数据处理方法。处理器通过运行存储在存储器中的非暂态软件程序以及程序，从而实现上述本申请实施例中的多媒体数据处理方法。
存储器可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储执行上述本申请实施例中的多媒体数据处理方法所需的数据等。此外，存储器可以包括高速随机存取存储器，还可以包括非暂态存储器，例如至少一个磁盘存储器件、闪存器件、或其他非暂态固态存储器件。在一些实施方式中，存储器包括相对于处理器远程设置的存储器，这些远程存储器可以通过网络连接至该多媒体 数据处理装置。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
实现上述本申请实施例中的多媒体数据处理方法所需的非暂态软件程序以及程序存储在存储器中，当被一个或者多个处理器执行时，执行上述本申请实施例中的多媒体数据处理方法，例如，执行以上描述的图1中的方法步骤S100至步骤S400，图2A中的方法步骤S2001至步骤S2004，图3A中的方法步骤S3001至步骤S3002，图4A中的方法步骤S2005至步骤S2007。获取多媒体数据的音频流和视频流；解析音频流，得到文本特征数据，根据预设的映射关系对照匹配文本特征数据以确定话题特征数据；解析视频流，得到表情特征数据，根据预设的映射关系对照匹配表情特征数据以确定情感指数；基于文本特征数据、情感指数和话题特征数据对多媒体数据进行渲染。基于此，本申请通过对多人商谈通话的多媒体信息进行解析提取、智能分析和渲染增强，在现有通话产品的语音和视频的基础上，挖掘出多维度的价值信息，包括语音文本、情感指数，并且将语音文本的内容根据预设的映射关系进一步二次加工，使得通话产品能够呈现更丰富智能的信息，同时，通过预设不同的映射关系，能够灵活的处理不同的商务场景和会谈内容，扩展技术方案的应用领域，可以有效的挖掘通话类产品的潜在价值，给通话商谈带来全新的体验，增加商谈业务使用者的会谈竞争力，提高决策效率，赢得更多的话语权，有效抓住商机。
此外，本申请实施例还提供了一种计算机可读存储介质，该计算机可读存储介质存储有计算机可执行程序，该计算机可执行程序被一个或多个控制处理器执行，例如，被图7中的一个处理器执行，可使得上述一个或多个处理器执行上述本申请实施例中的多媒体数据处理方法，例如，执行以上描述的图1中的方法步骤S100至步骤S400，图2A中的方法步骤S2001至步骤S2004，图3A中的方法步骤S3001至步骤S3002，图4A中的方法步骤S2005至步骤S2007。获取多媒体数据的音频流和视频流；解析音频流，得到文本特征数据，根据预设的映射关系对照匹配文本特征数据以确定话题特征数据；解析视频流，得到表情特征数据，根据预设的映射关系对照匹配表情特征数据以确定情感指数；基于文本特征数据、情感指数和话题特征数据对多媒体数据进行渲染。基于此，本申请通过对多人商谈通话的多媒体信息进行解析提取、智能分析和渲染增强，在现有通话产品的语音和视频的基础上，挖掘出多维度的价值信息，包括语音文本、情感指数，并且将语音文本的内容根据预设的映射关系进一步二次加工，使得通话产品能够呈现更丰富智能的信息，同时，通过预设不同的映射关系，能够灵活的处理不同的商务场景和会谈内容，扩展技术方案的应用领域，可以有效的挖掘通话类产品的潜在价值，给通话商谈带来全新的体验，增加商谈业务使用者的会谈竞争力，提高决策效率，赢得更多的话语权，有效抓住商机。
本领域普通技术人员可以理解，上文中所公开方法中的全部或某些步骤、系统可以被实施为软件、固件、硬件及其适当的组合。某些物理组件或所有物理组件可以被实施为由处理器，如中央处理器、数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读程序、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、 数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读程序、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。
以上是对本申请的部分实施进行说明，但本申请并不局限于上述实施方式，熟悉本领域的技术人员在不违背本申请本质的共享条件下还可作出种种等同的变形或替换，这些等同的变形或替换均包括在本申请权利要求所限定的范围内。

Claims (13)

  1. 一种多媒体数据处理方法，包括：
    获取多媒体数据的音频流和视频流；
    解析所述音频流，得到文本特征数据，根据预设的映射关系对照匹配所述文本特征数据以确定话题特征数据；
    解析所述视频流，得到表情特征数据，根据预设的映射关系对照匹配所述表情特征数据以确定情感指数；
    基于所述文本特征数据、所述情感指数和所述话题特征数据对所述多媒体数据进行渲染。
  2. 根据权利要求1所述的方法，其中，所述映射关系包括如下至少之一：
    预置微表情组合与情感指数的映射关系；或
    预置关键短句与场景模板的映射关系；或
    预置声明语句与命令序列的映射关系；或
    预置敏感语句与渲染集的映射关系。
  3. 根据权利要求2所述的方法，其中，所述解析所述音频流，得到文本特征数据，包括：
    对所述音频流进行频率特征学习，得到频率特征频段；
    利用所述频率特征频段对所述音频流过滤，得到若干特征音频流；
    对所述特征音频流进行语音识别，得到字幕文本；
    对所述特征音频流进行音强分析，得到音强值，当所述音强值达到预设阈值，则输出字幕附加文本。
  4. 根据权利要求3所述的方法，其中，所述解析所述视频流，得到表情特征数据，包括：
    将所述视频流输入到预置的深度学习模型，识别获得面部区域坐标；
    根据所述面部区域坐标，对所述视频流中面部区域的连续帧进行微表情识别分块，得到微表情组合。
  5. 根据权利要求4所述的方法，其中，所述根据预设的映射关系对照匹配所述表情特征数据以确定情感指数，包括：
    利用所述映射关系，将所述微表情组合与所述预置微表情组合进行对照匹配，得到对应的情感指数。
  6. 根据权利要求3所述的方法，其中，所述根据预设的映射关系对照匹配所述文本特征数据以确定话题特征数据，包括：
    将所述字幕文本通过自然语言处理，获得短句序列；
    利用所述映射关系，将所述短句序列与所述预置关键短句进行对照匹配，得到对应的所述场景模板；
    依据对应的所述场景模板，对所述字幕文本、所述字幕附加文本进行深层处理获得话题特征数据。
  7. 根据权利要求6所述的方法，其中，所述依据对应的所述场景模板，对所述字幕文本、所述字幕附加文本进行深层处理获得话题特征数据，包括：
    利用所述映射关系，将所述短句序列和所述字幕附加文本与所述预置声明语句进行对照匹配，得到对应的所述命令序列；
    执行所述命令序列，获得话题特征数据。
  8. 根据权利要求7所述的方法，其中，所述执行所述命令序列，获得话题特征数据，包括如下至少之一：
    当所述命令序列包含在线检索指令，则检索在线数据库，得到在线检索数据集；或
    当所述命令序列包含离线检索指令，则检索离线数据库，得到离线检索数据集；或
    当所述命令序列包含数据加工指令，则对数据进行二次加工，得到二次加工数据。
  9. 根据权利要求4至8任一所述的方法，其中，所述基于所述文本特征数据、所述情感指数和所述话题特征数据对所述多媒体数据进行渲染，包括：
    利用所述字幕文本，获得所述特征音频流启动时对应的声音时间序列；
    利用所述微表情组合，获得嘴型表情启动时对应的嘴型时间序列；
    判断所述声音时间序列和所述嘴型时间序列的一致性，将所述字幕文本、所述情感指数和所述面部区域坐标在时序上对齐；
    利用所述映射关系，将所述短句序列和所述预置敏感语句进行对照匹配，得到对应的所述渲染集；
    根据所述渲染集的配置，将所述字幕文本、所述情感指数和话题特征数据与所述视频流渲染。
  10. 根据权利要求9所述的方法，其中，所述渲染集的配置，包括如下至少之一：
    渲染坐标；或渲染时间；或渲染颜色主题。
  11. 一种多媒体数据处理装置，包括：
    音频处理模块，被配置为接收并解析音频流，获得文本特征数据；
    视频处理模块，被配置为接收并解析视频流，得到表情特征数据；
    映射关系模块，被配置为处理所述文本特征数据得到话题特征数据，处理所述表情特征数据得到情感指数；
    渲染模块，被配置为将所述文本特征数据、所述话题特征数据和所述情感指数与所述视频流渲染。
  12. 一种电子设备，包括：存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现如权利要求1至10中任意一项所述的多媒体数据处理方法。
  13. 一种计算机可读存储介质，存储有计算机可执行程序，所述计算机可执行程序用于使计算机执行如权利要求1至10任意一项所述的多媒体数据处理方法。
PCT/CN2023/101786 2022-06-24 2023-06-21 多媒体数据处理方法、装置和计算机可读存储介质 Ceased WO2023246888A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23826533.4A EP4543012A4 (en) 2022-06-24 2023-06-21 METHOD AND APPARATUS FOR PROCESSING MULTIMEDIA DATA, AND COMPUTER-READABLE STORAGE MEDIUM
US18/878,212 US20250384605A1 (en) 2022-06-24 2023-06-21 Multimedia data processing method and apparatus, and computer-readable storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202210722924.7A CN117319701A (zh) 2022-06-24 2022-06-24 多媒体数据处理方法、装置和计算机可读存储介质
CN202210722924.7 2022-06-24

Publications (1)

Publication Number Publication Date
WO2023246888A1 true WO2023246888A1 (zh) 2023-12-28

Family

ID=89235961

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/101786 Ceased WO2023246888A1 (zh) 2022-06-24 2023-06-21 多媒体数据处理方法、装置和计算机可读存储介质

Country Status (4)

Country Link
US (1) US20250384605A1 (zh)
EP (1) EP4543012A4 (zh)
CN (1) CN117319701A (zh)
WO (1) WO2023246888A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119110023A (zh) * 2024-11-08 2024-12-10 广州兴趣岛信息科技有限公司 一种ai自动拨打通话处理方法及系统

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20250008290A1 (en) * 2023-06-27 2025-01-02 University Of Central Florida Research Foundation, Inc. Spatially Explicit Auditory Cues for Enhanced Situational Awareness
CN119402699A (zh) * 2024-10-31 2025-02-07 北京字跳网络技术有限公司 视频处理方法、装置、设备及介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103365959A (zh) * 2013-06-03 2013-10-23 深圳市爱渡飞科技有限公司 一种语音搜索方法及装置
CN106161873A (zh) * 2015-04-28 2016-11-23 天脉聚源（北京）科技有限公司 一种视频信息提取推送方法及系统
CN109886258A (zh) * 2019-02-19 2019-06-14 新华网(北京)科技有限公司 提供多媒体信息的关联信息的方法、装置及电子设备
CN110365933A (zh) * 2019-05-21 2019-10-22 武汉兴图新科电子股份有限公司 一种基于ai的视频会议会议纪要在线生成装置及方法
CN110414465A (zh) * 2019-08-05 2019-11-05 北京深醒科技有限公司 一种视频通讯的情感分析方法
CN114095782A (zh) * 2021-11-12 2022-02-25 广州博冠信息科技有限公司 一种视频处理方法、装置、计算机设备及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10049263B2 (en) * 2016-06-15 2018-08-14 Stephan Hau Computer-based micro-expression analysis
US10791078B2 (en) * 2017-07-30 2020-09-29 Google Llc Assistance during audio and video calls
US20210076002A1 (en) * 2017-09-11 2021-03-11 Michael H Peters Enhanced video conference management
CN110798636B (zh) * 2019-10-18 2022-10-11 腾讯数码（天津）有限公司 字幕生成方法及装置、电子设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103365959A (zh) * 2013-06-03 2013-10-23 深圳市爱渡飞科技有限公司 一种语音搜索方法及装置
CN106161873A (zh) * 2015-04-28 2016-11-23 天脉聚源（北京）科技有限公司 一种视频信息提取推送方法及系统
CN109886258A (zh) * 2019-02-19 2019-06-14 新华网(北京)科技有限公司 提供多媒体信息的关联信息的方法、装置及电子设备
CN110365933A (zh) * 2019-05-21 2019-10-22 武汉兴图新科电子股份有限公司 一种基于ai的视频会议会议纪要在线生成装置及方法
CN110414465A (zh) * 2019-08-05 2019-11-05 北京深醒科技有限公司 一种视频通讯的情感分析方法
CN114095782A (zh) * 2021-11-12 2022-02-25 广州博冠信息科技有限公司 一种视频处理方法、装置、计算机设备及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4543012A4 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119110023A (zh) * 2024-11-08 2024-12-10 广州兴趣岛信息科技有限公司 一种ai自动拨打通话处理方法及系统

Also Published As

Publication number Publication date
US20250384605A1 (en) 2025-12-18
EP4543012A1 (en) 2025-04-23
CN117319701A (zh) 2023-12-29
EP4543012A4 (en) 2025-09-10

Similar Documents

Publication Publication Date Title
Gleason et al. Making memes accessible
Ou et al. Multimodal local-global attention network for affective video content analysis
EP4543012A1 (en) Multimedia data processing method and apparatus, and computer-readable storage medium
CN117252259A (zh) 基于深度学习的自然语言理解方法及ai助教系统
Baur et al. eXplainable cooperative machine learning with NOVA
US10062384B1 (en) Analysis of content written on a board
CN119904901A (zh) 基于大模型的情绪识别方法以及相关装置
US12288379B1 (en) Common sense reasoning for deepfake detection
CN116977992A (zh) 文本信息识别方法、装置、计算机设备和存储介质
Wang et al. Automatic Chinese meme generation using deep neural networks
CN119831057A (zh) 基于情感识别的ai对话系统
CN119402728A (zh) 基于文本的视频生成方法和装置、电子设备、存储介质
CN112084788A (zh) 一种影像字幕隐式情感倾向自动标注方法及系统
CN114741472B (zh) 辅助绘本阅读的方法、装置、计算机设备及存储介质
Diamantini et al. Automatic annotation of corpora for emotion recognition through facial expressions analysis
CN119049051B (zh) 一种基于文本数据的图像生成方法、系统及装置
CN118916707A (zh) 多模态人格分析方法、分析模型、可读存储介质及装置
CN118428340A (zh) 作文批改方法、装置、电子设备及存储介质
CN116842218A (zh) 全自动生成视频会议纪要的方法以及存储介质
Nivethika et al. Roberta-Powered Sentiment Detection in Natural Language Processing
Letaifa et al. The CG-MER dyadic multimodal dataset for spontaneous french conversations: annotation, analysis and assessment benchmark
Lakshmi et al. Virtual Caricature Assistant using Voice Analysis and
Poornima et al. Police sketch generator using generative adversarial networks
CN120950901B (zh) 基于多模态数据的员工情绪指数评测方法以及装置
Punj et al. Detection of emotions with deep learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23826533

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 18878212

Country of ref document: US

WWE Wipo information: entry into national phase

Ref document number: 2023826533

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2023826533

Country of ref document: EP

Effective date: 20250120

WWP Wipo information: published in national office

Ref document number: 2023826533

Country of ref document: EP

WWP Wipo information: published in national office

Ref document number: 18878212

Country of ref document: US