WO2024083121A1 - 一种数据处理方法及其装置 - Google Patents

一种数据处理方法及其装置 Download PDF

Info

Publication number
WO2024083121A1
WO2024083121A1 PCT/CN2023/124977 CN2023124977W WO2024083121A1 WO 2024083121 A1 WO2024083121 A1 WO 2024083121A1 CN 2023124977 W CN2023124977 W CN 2023124977W WO 2024083121 A1 WO2024083121 A1 WO 2024083121A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
text
feature
target object
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2023/124977
Other languages
English (en)
French (fr)
Inventor
朱艺
刘健庄
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP23879109.9A priority Critical patent/EP4592866B1/en
Publication of WO2024083121A1 publication Critical patent/WO2024083121A1/zh
Priority to US19/182,947 priority patent/US20250272945A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/253—Fusion techniques of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/12—Use of codes for handling textual entities
    • G06F40/126—Character encoding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/30—Semantic analysis
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/14—Image acquisition
    • G06V30/148—Segmentation of character regions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/10—Character recognition
    • G06V30/19—Recognition using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
    • G06V30/40—Document-oriented image-based pattern recognition
    • G06V30/41—Analysis of document content

Definitions

  • the present application relates to the field of artificial intelligence, and in particular to a data processing method and device thereof.
  • Artificial Intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
  • artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.
  • Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.
  • Language-driven precise instance segmentation is a special semantic segmentation technology. It refers to accurately segmenting the instance targets described by language in an image according to the guidance of natural language. Its characteristics are: 1) Traditional semantic segmentation models predict the same label for all targets belonging to the same category and do not distinguish different targets in the same category, while language-driven precise instance segmentation needs to accurately identify an instance target corresponding to the language description from multiple similar targets; 2) The semantic segmentation model needs to pre-define a set of semantic category labels in order to learn to segment targets of these categories, while language-driven precise instance segmentation can accept more flexible natural language input and is not limited to target categories.
  • the present application provides a data processing method that can effectively solve the problems of inaccurate target positioning and mask prediction or detection box prediction in existing language-driven precise instance segmentation methods, thereby improving the processing accuracy of the model.
  • the present application provides a data processing method, comprising: obtaining a first image feature corresponding to an image and a text feature corresponding to a text; the semantics of the text corresponds to a target object, and the text indicates a region corresponding to the target object predicted from the image; based on a plurality of preset first embedding vectors and the first image feature, a plurality of second embedding vectors are obtained through a neural network, each second embedding vector corresponding to an object in the image; each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature; based on the similarity between the text feature and the plurality of second embedding vectors, a weight corresponding to each second embedding vector is determined, and the plurality of weights are used to fuse (for example, weighted) with the plurality of second image features to determine a predicted region corresponding to the target object.
  • the image may include multiple objects including the target object, each second embedding vector corresponds to an object in the image, and one or more embedding vectors in the multiple second embedding vectors may correspond to the target object.
  • corresponding here can be understood as the second embedding vector is used to describe the characteristics of an object in the image, and the second embedding vector obtained by the neural network can distinguish different objects in the image, so that the subsequent prediction process can be based on the object granularity.
  • This is equivalent to changing the image features from pixel granularity to target object granularity, that is, introducing target integrity constraints in cross-modal feature fusion, fusing pixels belonging to the same target as a whole with language encoding, and activating instance areas based on targets.
  • This can effectively solve the problems of inaccurate target positioning and mask prediction or detection box prediction in existing language-driven precise instance segmentation methods, thereby improving the processing accuracy of the model.
  • the predicted area is a mask area or a detection box.
  • the semantics of the text corresponds to the target object, specifically including: the semantics of the text is used to describe the characteristics of the target object.
  • acquiring a first image feature corresponding to the image and a text feature corresponding to the text includes:
  • the third image feature and the first text feature are fused through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • the first image feature is a feature that is upsampled to a size consistent with the image.
  • the neural grid includes multiple transformer layers.
  • a data processing method includes:
  • each second embedding vector corresponds to an object in the image
  • each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature
  • the feature extraction network and the neural network are updated according to the difference between the predicted area and the real area corresponding to the target object in the image.
  • the image may include multiple objects including the target object, each second embedding vector corresponds to an object in the image, and one or more embedding vectors in the multiple second embedding vectors may correspond to the target object.
  • corresponding here can be understood as the second embedding vector is used to describe the characteristics of an object in the image, and the second embedding vector obtained by the neural network can distinguish different objects in the image, so that the subsequent prediction process can be based on the object granularity.
  • This is equivalent to changing the image features from pixel granularity to target object granularity, that is, introducing target integrity constraints in cross-modal feature fusion, fusing pixels belonging to the same target as a whole with language encoding, and activating instance areas based on targets.
  • This can effectively solve the problems of inaccurate target positioning and mask prediction or detection box prediction in existing language-driven precise instance segmentation methods, thereby improving the processing accuracy of the model.
  • the predicted area is a mask area or a detection box.
  • the semantics of the text corresponds to the target object, specifically including: the semantics of the text is used to describe the characteristics of the target object.
  • acquiring a first image feature corresponding to the image and a text feature corresponding to the text includes:
  • the third image feature and the first text feature are fused through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • the present application provides a data processing device, including:
  • a processing module configured to obtain a first image feature corresponding to the image and a text feature corresponding to the text; the semantics of the text corresponds to the target object, and the text indicates a region corresponding to the target object predicted from the image;
  • multiple second embedding vectors are obtained through a neural network, each of which a second embedding vector corresponding to an object in the image; each of the second embedding vectors and the first image feature is used to fuse to obtain a corresponding second image feature;
  • a weight corresponding to each of the second embedding vectors is determined, and the multiple weights are used to be fused with the multiple second image features to determine the predicted area corresponding to the target object.
  • the image may include multiple objects including the target object, each second embedding vector corresponds to an object in the image, and one or more embedding vectors in the multiple second embedding vectors may correspond to the target object.
  • the "correspondence" here can be understood as the second embedding vector is used to describe the characteristics of an object in the image, and the second embedding vector obtained by the neural network can distinguish different objects in the image, so that the subsequent prediction process can be based on the object granularity.
  • This is equivalent to changing the image features from pixel granularity to target object granularity, that is, introducing target integrity constraints in cross-modal feature fusion, fusing pixels belonging to the same target as a whole with language encoding, and activating instance areas based on targets.
  • This can effectively solve the problems of inaccurate target positioning and mask prediction or detection box prediction in existing language-driven precise instance segmentation methods, thereby improving the processing accuracy of the model.
  • the predicted area is a mask area or a detection box.
  • the semantics of the text corresponds to the target object, specifically including: the semantics of the text is used to describe the characteristics of the target object.
  • the processing module is specifically configured to:
  • the third image feature and the first text feature are fused through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • the first image feature is a feature that is upsampled to a size consistent with the image.
  • the neural grid includes multiple transformer layers.
  • the present application provides a data processing device, comprising:
  • a processing module used for obtaining a first image feature corresponding to an image and a text feature corresponding to a text; the semantics of the text corresponds to a target object, and the text indicates a region corresponding to the target object predicted from the image; the first image feature and the text feature are obtained according to a feature extraction network;
  • each second embedding vector corresponds to an object in the image
  • each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature
  • An updating module is used to update the feature extraction network and the neural network according to the difference between the predicted area and the real area corresponding to the target object in the image.
  • the predicted area is a mask area or a detection box.
  • the semantics of the text corresponds to the target object, specifically including: the semantics of the text is used to describe the characteristics of the target object.
  • the processing module is specifically configured to:
  • the third image feature and the first text feature are fused through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • an embodiment of the present application provides a data processing device, which may include a memory, a processor, and a bus system, wherein the memory is used to store programs, and the processor is used to execute the programs in the memory to execute the above-mentioned first aspect and any optional method thereof, and the above-mentioned second aspect and any optional method thereof.
  • an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored.
  • the computer-readable storage medium When the computer-readable storage medium is run on a computer, the computer executes the above-mentioned first aspect and any optional method thereof, and the above-mentioned second aspect and any optional method thereof.
  • an embodiment of the present application provides a computer program which, when executed on a computer, enables the computer to execute the above-mentioned first aspect and any optional method thereof, and the above-mentioned second aspect and any optional method thereof.
  • the present application provides a chip system, which includes a processor for supporting the execution of a data processing device to implement the functions involved in the above aspects, such as sending or processing the data involved in the above methods; or information.
  • the chip system also includes a memory, which is used to store program instructions and data necessary for the execution device or training device.
  • the chip system can be composed of chips, or it can include chips and other discrete devices.
  • FIG1A is a schematic diagram of a structure of an artificial intelligence main framework
  • FIG. 1B to FIG. 1C are schematic diagrams of the application system framework of the present application.
  • FIG1D is a schematic diagram of an optional hardware structure of a terminal
  • FIG2 is a schematic diagram of the structure of a server
  • FIG3 is a schematic diagram of a system architecture of the present application.
  • FIG4 is a process of a cloud service
  • FIG5 is a schematic diagram of a network structure
  • FIG6 is a flowchart of a data processing method provided in an embodiment of the present application.
  • FIGS. 7 to 10 are schematic diagrams of a process of a data processing method provided in an embodiment of the present application.
  • FIG. 11A and FIG. 11B are schematic diagrams of a beneficial effect of the present application.
  • FIG12 is a schematic diagram of a structure of a data processing device provided in an embodiment of the present application.
  • FIG13 is a schematic diagram of a structure of an execution device provided in an embodiment of the present application.
  • FIG14 is a schematic diagram of a structure of a training device provided in an embodiment of the present application.
  • FIG. 15 is a schematic diagram of the structure of a chip provided in an embodiment of the present application.
  • the terms “substantially,””about,” and similar terms are used as terms of approximation rather than as terms of degree, and are intended to take into account the inherent variations in measurements or calculations that will be known to those of ordinary skill in the art.
  • the use of “may” when referring to embodiments means “one or more possible embodiments.”
  • the terms “use,””using,” and “used” may be considered synonymous with the terms “utilize,””utilizing,” and “utilized,” respectively.
  • the term “exemplary” is intended to refer to an example or illustration.
  • Figure 1A shows a structural diagram of the main framework of artificial intelligence.
  • the following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain” (horizontal axis) and “IT value chain” (vertical axis).
  • the "intelligent information chain” reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom".
  • the "IT value chain” reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.
  • the infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and is supported by the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.
  • smart chips CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips
  • the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc.
  • sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.
  • the data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence.
  • the data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.
  • Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.
  • machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
  • Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
  • Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.
  • some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
  • Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.
  • the present application can be applied to the field of image processing in the field of artificial intelligence. Taking image processing as an example, multiple application scenarios of products will be introduced below.
  • image processing can be used as a core algorithm module of the robot's visual language navigation system.
  • a user wants to command a household robot to walk to a chair and pick up a vase through language instructions.
  • the robot needs to accurately segment or detect the target instances of the chair and the vase described in the language before it can complete the task of picking up the vase.
  • image processing functions can be applied to autonomous driving platforms.
  • the recognition module of the intelligent driving system needs to first understand the user's natural language instructions and accurately segment or detect the yellow car to complete the user's request.
  • image processing functions can be applied to interactive image editing systems, which need to modify images based on the user's natural language description.
  • the image processing function can locate the area that the user wants to modify, and then combine it with existing image editing tools to modify the image content.
  • image processing applications applications with image processing functions (hereinafter referred to as image processing applications) or cloud services provided by cloud-side servers, etc., which are introduced below:
  • the product form of the embodiment of the present application can be an image processing application, in particular, a language-driven image processing application.
  • the language-driven image processing application can be run on a terminal device or a server on the cloud side.
  • a language-driven image processing application can perform tasks such as image segmentation or target detection based on the input image and text to obtain processing results.
  • the processing results can be image segmentation results (mask area) and detection boxes.
  • the image segmentation results (mask area) and the detection box can contain objects indicated by the semantics of the text (such as the target object in the embodiment of the present application).
  • a user can open an image processing application installed on a terminal device and input images and text.
  • the image processing application can process the image and text using the method provided in an embodiment of the present application and present the processing results to the user (the presentation method may be but is not limited to display, saving, uploading to the cloud, etc.).
  • a user can open an image processing application installed on a terminal device and input an image.
  • the image processing application can send the image to a server on the cloud side.
  • the server on the cloud side processes the image and text using the method provided in an embodiment of the present application and transmits the processing result back to the terminal device.
  • the terminal device can present the processing result to the user (the presentation method may be but is not limited to display, saving, uploading to the cloud side, etc.).
  • FIG. 1B is a schematic diagram of the functional architecture of an image processing application in an embodiment of the present application:
  • an image processing application 102 may receive input parameters 101 (e.g., including an image) and generate a processing result 103.
  • the image processing application 102 may be executed on (for example) at least one computer system and includes computer codes, which, when executed by one or more computers, cause the computers to execute the method provided in the embodiments of the present application.
  • FIG. 1C is a schematic diagram of the physical architecture for running an image processing application in an embodiment of the present application:
  • FIG. 1C shows a schematic diagram of a system architecture.
  • the system may include a terminal 100 and a server 200.
  • the server 200 may include one or more servers (FIG. 1C is illustrated by taking one server as an example), and the server 200 may provide the method provided in the embodiment of the present application for one or more terminals.
  • an image processing application can be installed on the terminal 100.
  • the above application and web page can provide an interface.
  • the terminal 100 can receive relevant parameters entered by the user on the language-driven image processing interface and send the above parameters to the server 200.
  • the server 200 can obtain the processing results based on the received parameters and return the processing results to the terminal 100.
  • the terminal 100 can also complete the action of obtaining the processing result based on the received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited to this.
  • the terminal 100 in the embodiment of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR)/virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiment of the present application does not impose any limitation on this.
  • AR augmented reality
  • VR virtual reality
  • UMPC ultra-mobile personal computer
  • PDA personal digital assistant
  • FIG. 1D shows a schematic diagram of an optional hardware structure of the terminal 100 .
  • the terminal 100 may include components such as a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, and a power supply 190.
  • a radio frequency unit 110 such as a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, and a power supply 190.
  • FIG. 1D is merely an example of a terminal or a multi-function device, and does not constitute a limitation on the terminal or the multi-function device, and may include more or less components than shown in the figure, or combine certain components, or different components.
  • the input unit 130 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the portable multifunctional device.
  • the input unit 130 may include a touch screen 131 (optional) and/or other input devices 132.
  • the touch screen 131 can collect user touch operations on or near it (such as operations performed by the user using fingers, joints, stylus, or any other suitable object on or near the touch screen), and drive the corresponding connection device according to a pre-set program.
  • the touch screen can detect the user's touch operation. For the touch action on the touch screen, the touch action is converted into a touch signal and sent to the processor 170, and the command sent by the processor 170 can be received and executed; the touch signal at least includes the touch point coordinate information.
  • the touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user.
  • the touch screen can be implemented in various types such as resistive, capacitive, infrared and surface acoustic wave.
  • the input unit 130 can also include other input devices.
  • other input devices 132 can include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.
  • the input device 132 may receive input images, texts, and the like.
  • the display unit 140 may be used to display information input by the user or provided to the user, various menus of the terminal 100, interactive interfaces, file display, and/or playback of any multimedia file.
  • the display unit 140 may be used to display the interface of an image processing application, processing results, etc.
  • the memory 120 can be used to store instructions and data.
  • the memory 120 can mainly include an instruction storage area and a data storage area.
  • the data storage area can store various data, such as multimedia files, texts, etc.;
  • the instruction storage area can store software units such as operating systems, applications, instructions required for at least one function, or their subsets and extensions. It can also include non-volatile random access memory; provide the processor 170 with hardware, software and data resources including management of computing and processing equipment, and support control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.
  • the processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal 100. By running or executing instructions stored in the memory 120 and calling data stored in the memory 120, it executes various functions of the terminal 100 and processes data, thereby controlling the terminal device as a whole.
  • the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 170.
  • the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips.
  • the processor 170 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.
  • the memory 120 can be used to store software codes related to the data processing method
  • the processor 170 can execute the steps of the chip data processing method, and can also schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.
  • the RF unit 110 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 170 for processing; in addition, the designed uplink data is sent to the base station.
  • the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc.
  • the RF unit 110 can also communicate with network devices and other devices through wireless communication.
  • the wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.
  • GSM Global System of Mobile communication
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • LTE Long Term Evolution
  • email Short Messaging Service
  • the RF unit 110 can send the image to the server 200 and receive the processing result sent by the server 200.
  • radio frequency unit 110 is optional and can be replaced by other communication interfaces, such as a network port.
  • the terminal 100 also includes a power supply 190 (such as a battery) for supplying power to various components.
  • a power supply 190 such as a battery
  • the power supply can be logically connected to the processor 170 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.
  • the terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector. It can be used to connect the terminal 100 to communicate with other devices, and can also be used to connect a charger to charge the terminal 100.
  • an external interface 180 can be a standard Micro USB interface or a multi-pin connector. It can be used to connect the terminal 100 to communicate with other devices, and can also be used to connect a charger to charge the terminal 100.
  • the terminal 100 may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which are not described in detail here. Some or all of the methods described below may be applied to the terminal 100 as shown in FIG. 1D .
  • WiFi wireless fidelity
  • Bluetooth Bluetooth
  • FIG2 provides a schematic diagram of the structure of a server 200.
  • the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204.
  • the processor 202, the memory 204, and the communication interface 203 communicate with each other via the bus 201.
  • the bus 201 may be a peripheral component interconnect (PCI) bus or an extended industry standard bus.
  • PCI peripheral component interconnect
  • the bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in FIG2 , but it does not mean that there is only one bus or one type of bus.
  • the processor 202 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
  • CPU central processing unit
  • GPU graphics processing unit
  • MP microprocessor
  • DSP digital signal processor
  • the memory 204 may include a volatile memory (volatile memory), such as a random access memory (RAM).
  • volatile memory such as a random access memory (RAM).
  • RAM random access memory
  • non-volatile memory non-volatile memory
  • ROM read-only memory
  • flash memory flash memory
  • HDD hard drive
  • SSD solid state drive
  • the memory 204 may be used to store software codes related to the data processing method, and the processor 202 may execute the steps of the data processing method of the chip, and may also schedule other units to implement corresponding functions.
  • the above-mentioned terminal 100 and server 200 can be centralized or distributed devices, and the processors in the above-mentioned terminal 100 and server 200 (such as processor 170 and processor 202) can be hardware circuits (such as application specific integrated circuit (ASIC), field-programmable gate array (FPGA), general-purpose processor, digital signal processor (DSP), microprocessor or microcontroller, etc.), or a combination of these hardware circuits.
  • the processor can be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.
  • the steps related to the model reasoning process in the embodiments of the present application involve AI-related operations.
  • the instruction execution architecture of the terminal device and the server is not limited to the processor combined with the memory architecture described above.
  • the system architecture provided in the embodiments of the present application is described in detail below in conjunction with Figure 3.
  • FIG3 is a schematic diagram of a system architecture provided by an embodiment of the present application.
  • a system architecture 500 includes an execution device 510 , a training device 520 , a database 530 , a client device 540 , a data storage system 550 , and a data acquisition system 560 .
  • the execution device 510 includes a calculation module 511, an I/O interface 512, a preprocessing module 513 and a preprocessing module 514.
  • the calculation module 511 may include a target model/rule 501, and the preprocessing module 513 and the preprocessing module 514 are optional.
  • the execution device 510 may be a terminal device or a server that runs the above-mentioned image processing application.
  • the data acquisition device 560 is used to acquire training samples.
  • the training samples may be multiple images, etc.
  • the data acquisition device 560 stores the training samples in the database 530 .
  • the training device 520 can train the neural network to be trained (such as the cross-modal language model in the embodiment of the present application (such as a text encoder, an image encoder, a target encoder, etc.)) based on the training samples maintained in the database 530 to obtain the target model/rule 501.
  • the neural network such as the cross-modal language model in the embodiment of the present application (such as a text encoder, an image encoder, a target encoder, etc.)
  • the neural network to be trained such as the cross-modal language model in the embodiment of the present application (such as a text encoder, an image encoder, a target encoder, etc.)
  • the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.
  • the training samples maintained in the database 530 may not all come from the data acquisition device 560, but may also be received from other devices. It should also be noted that the training device 520 may not train the target model/rule 501 entirely based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training. The above description should not be used as a limitation on the embodiments of the present application.
  • the target model/rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG3 .
  • the execution device 510 can be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR)/virtual reality (VR) device, a vehicle terminal, etc., and can also be a server, etc.
  • AR augmented reality
  • VR virtual reality
  • the training device 520 may transfer the trained model to the execution device 510 .
  • the execution device 510 is configured with an input/output (I/O) interface 512 for data interaction with an external device.
  • the user can input data (such as images in the embodiments of the present application) to the I/O interface 512 through the client device 540.
  • the preprocessing module 513 and the preprocessing module 514 are used to preprocess the input data received by the I/O interface 512. It should be understood that there may be no preprocessing module 513 and the preprocessing module 514 or only one preprocessing module. When there is no preprocessing module 513 and the preprocessing module 514, the computing module 511 may be directly used to process the input data.
  • the execution device 510 When the execution device 510 pre-processes the input data, or when the calculation module 511 of the execution device 510 performs calculations and other related processing, the execution device 510 can call the data, code, etc. in the data storage system 550 for the corresponding processing, and can also store the corresponding processing data. The processed data, instructions, etc. are stored in the data storage system 550.
  • the I/O interface 512 provides the processing results to the client device 540 and thus to the user.
  • the user can manually give input data, and the “manually given input data” can be operated through the interface provided by the I/O interface 512.
  • the client device 540 can automatically send input data to the I/O interface 512. If the client device 540 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 540. The user can view the results output by the execution device 510 on the client device 540, and the specific presentation form can be a specific method such as display, sound, action, etc.
  • the client device 540 can also be used as a data acquisition terminal to collect the input data of the input I/O interface 512 and the output results of the output I/O interface 512 as shown in the figure as new sample data, and store them in the database 530.
  • the I/O interface 512 directly stores the input data of the input I/O interface 512 and the output results of the output I/O interface 512 as new sample data in the database 530.
  • FIG. 3 is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation.
  • the data storage system 550 is an external memory relative to the execution device 510. In other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above-mentioned execution device 510 can be deployed in the client device 540.
  • the computing module 511 of the above-mentioned execution device 520 can obtain the code stored in the data storage system 550 to implement the steps related to the model reasoning process in the embodiment of the present application.
  • the computing module 511 of the execution device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits.
  • the training device 520 may be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the above-mentioned hardware systems without an execution instruction function and hardware systems with an execution instruction function.
  • the computing module 511 of the execution device 520 can be a hardware system with an execution instruction function, and the steps related to the model reasoning process provided in the embodiment of the present application can be software codes stored in the memory.
  • the computing module 511 of the execution device 520 can obtain the software code from the memory and execute the obtained software code to implement the steps related to the model reasoning process provided in the embodiment of the present application.
  • the computing module 511 of the execution device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model reasoning process provided in the embodiments of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the computing module 511 of the execution device 520, which is not limited here.
  • the above-mentioned training device 520 can obtain the code stored in the memory (not shown in Figure 3, which can be integrated into the training device 520 or deployed separately from the training device 520) to implement the steps related to model training in an embodiment of the present application.
  • the training device 520 may include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits.
  • the training device 520 may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.
  • the training device 520 can be a combination of a hardware system that does not have the function of executing instructions and a hardware system that has the function of executing instructions. Some of the steps related to the model training provided in the embodiments of the present application can also be implemented by the hardware system that does not have the function of executing instructions in the training device 520, which is not limited here.
  • the server can provide language-driven image processing services to the end side through an application programming interface (API).
  • API application programming interface
  • the terminal device can send relevant parameters (such as images) to the server through the API provided by the cloud, and the server can Based on the received parameters, the processing results are obtained, etc.), and the processing results are returned to the terminal.
  • relevant parameters such as images
  • the server can Based on the received parameters, the processing results are obtained, etc.
  • FIG. 4 shows a process of using a language-driven image processing cloud service provided by a cloud platform.
  • SDK software development kit
  • the cloud platform provides multiple development versions of the SDK for users to choose according to the requirements of the development environment, such as JAVA version SDK, Python version SDK, PHP version SDK, Android version SDK, etc.
  • the SDK project is imported into the local development environment, and configuration and debugging are performed in the local development environment.
  • the local development environment can also be used to develop other functions, thus forming an application that integrates language-driven image processing capabilities.
  • the language-driven image processing API call can be triggered.
  • the application triggers the language-driven image processing function an API request is initiated to the running instance of the language-driven image processing service in the cloud environment, wherein the API request carries an image, and the running instance in the cloud environment processes the image to obtain the processing result.
  • the cloud environment returns the processing results to the application, thereby completing a method call provided in an embodiment of the present application.
  • a neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:
  • n is a natural number greater than 1
  • Ws is the weight of xs
  • b is the bias of the neural unit.
  • f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal.
  • the output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function.
  • a neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit.
  • the input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field.
  • the local receptive field can be an area composed of several neural units.
  • FIG5 is a schematic diagram of the architecture of a transformer layer.
  • the neural network includes an embedding layer and at least one transformer layer, and the at least one transformer layer can be N transformer layers (N is an integer greater than 0), wherein each transformer layer includes an attention layer, an add&norm layer, a feed forward layer, and an add&norm layer that are adjacent in sequence.
  • the current input is embedded to obtain multiple embedding vectors;
  • P input vectors are obtained from the previous layer of the first transformer layer, and the first input vector among the P input vectors is taken as the center, and the intermediate vector corresponding to the first input vector is obtained based on the correlation between each input vector within the preset attention window range and the first input vector, so as to determine the P intermediate vectors corresponding to the P input vectors;
  • the P intermediate vectors are merged into Q output vectors, wherein the multiple output vectors obtained by the last transformer layer in the transformer layer are used as the feature representation of the current input.
  • the attention mechanism imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensations to increase the observation precision of some areas, and can use limited attention resources to quickly filter out high-value information from a large amount of information.
  • the attention mechanism can quickly extract important features of sparse data, and is therefore widely used in natural language processing tasks, especially machine translation.
  • the self-attention mechanism is an improvement on the attention mechanism, which reduces dependence on external information and is better at capturing the internal correlation of data or features.
  • the essential idea of the attention mechanism can be rewritten as the following formula:
  • Lx
  • represents the length of Source.
  • the formula means that the elements in Source are imagined as a series of data. For the composition, given an element Query in the target Target, by calculating the similarity or correlation between Query and each Key, the weight coefficient of the Value corresponding to each Key is obtained, and then the Value is weighted and summed to obtain the final Attention value. So in essence, the Attention mechanism is to perform a weighted summation of the Value values of the elements in the Source, and Query and Key are used to calculate the weight coefficient of the corresponding Value.
  • Attention can be understood as selectively screening out a small amount of important information from a large amount of information and focusing on these important information, ignoring most of the unimportant information.
  • the process of focusing is reflected in the calculation of the weight coefficient.
  • the self-attention mechanism can be understood as internal Attention (intra attention).
  • the Attention mechanism occurs between the Query element of the Target and all the elements in the Source.
  • the specific calculation process is the same, but the calculation object has changed.
  • NLP Natural language processing
  • Natural language is human language, and natural language processing (NLP) is the processing of human language. Natural language processing is the process of systematically analyzing, understanding, and extracting information from text data in an intelligent and efficient way.
  • NLP natural language processing
  • MT machine translation
  • NER named entity recognition
  • RE relation extraction
  • IE information extraction
  • sentiment analysis speech recognition
  • question answering and topic segmentation, etc.
  • the pre-trained language model is a natural language sequence encoder that encodes each word in a natural language sequence into a vector representation for prediction tasks. Its training consists of two stages. In the pre-training stage, the model is trained on language model tasks on large-scale unsupervised text to learn a word representation. In the fine-tuning stage, the model is initialized using the parameters learned in the pre-training stage, and is trained on downstream tasks such as text classification and sequence labeling with fewer steps, so that the semantic information obtained from pre-training can be successfully transferred to downstream tasks.
  • Convolutional neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller.
  • BP error back propagation
  • the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges.
  • the back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the super-resolution model, such as the weight matrix.
  • Language-driven precise instance segmentation is a special semantic segmentation technology. It refers to accurately segmenting the instance targets described by language in an image according to the guidance of natural language. Its characteristics are: 1) Traditional semantic segmentation models predict the same label for all targets belonging to the same category and do not distinguish different targets in the same category, while language-driven precise instance segmentation needs to accurately identify an instance target corresponding to the language description from multiple similar targets; 2) The semantic segmentation model needs to pre-define a set of semantic category labels in order to learn to segment targets of these categories, while language-driven precise instance segmentation can accept more flexible natural language input and is not limited to target categories.
  • the present application provides a data processing method.
  • the data processing method of the present application is described in detail below with reference to the accompanying drawings.
  • FIG. 6 is a flow chart of a data processing method provided in an embodiment of the present application.
  • a data processing method provided in an embodiment of the present application may include steps 601 to 603, and these steps are described in detail below.
  • the semantics of the text may indicate determining a mask region corresponding to a target object in an image (image segmentation task) or a detection box (object detection task).
  • the semantics of the text is used to describe the features of the target object. For example, if the image includes two vases, one red and one yellow, the text may be a red vase. For example, if the image includes two vases, one on the left and one on the right of the image, the text may be a vase on the left.
  • feature extraction and alignment may be performed on the image and the text to obtain a first image feature corresponding to the image and a text feature corresponding to the text.
  • obtaining the first image feature corresponding to the image and the text feature corresponding to the text specifically includes: processing the image through an image encoder to obtain the image feature corresponding to the image, processing the text through a text encoder to obtain the first text feature corresponding to the text, and fusing the third image feature and the first text feature through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • the image encoder f v (or visual encoder) can use Swin Transformer as a visual encoder to extract multi-level visual features for a given visual image I, and convert the multi-scale visual features generated by multiple stages (taking four stages as an example) in Swin Transformer into
  • the text encoder fl can use a multi-layer BERT (taking 12 layers as an example) as a language encoder.
  • the bidirectional attention mechanism can be implemented using the Word-pixel alignment module.
  • the Word-pixel alignment module is a cross-modal thresholded bidirectional attention module that aligns visual and language features in the feature space during the encoding phase of images and sentences.
  • the learnable feature threshold mechanism is to prevent the original feature information from being overwhelmed when updating the fused features.
  • the alignment effect of the Word-pixel alignment can be shown in Figure 8.
  • the language information is integrated into the visual encoding during the visual and language information encoding phase, and the visual information is integrated into the language encoding, so that the word features of the sentence and the corresponding pixel features in the picture are correlated in the cross-modal feature space.
  • the cross-modal bidirectional attention module BiAttn interacts visual and language features in the feature space.
  • This module is used to fuse the visual and language features of each stage of the dual encoder.
  • its operation is defined as follows:
  • V′ i ,L′ i BiAttn(V i ,L i ),i ⁇ 1,...,4 ⁇ ;
  • d k is the dimension of the joint visual language embedding space
  • W v , W l , W′ v , W′ l are all projection matrices.
  • Fi represents the fused features from the BCA module
  • F′i represents the suppressed fused features
  • ⁇ represents the matrix element-by-element multiplication.
  • MLP is a two-layer perceptron, the first layer is a linear layer, followed by a ReLU activation function, and the second layer is a linear layer, followed by a hyperbolic tangent activation function.
  • a multi-head attention layer can be used to fuse high-level features from the visual language encoder.
  • the high-level visual features V o and language features L o are projected into the same feature space, and then they are concatenated into a fused feature F o , which is then sent to the cross-attention layer.
  • a learnable position vector ep is added to the projected visual features.
  • the cross-attention layer outputs the feature S o .
  • a segmentation head in order to upsample the pixel-level features to the original image size to obtain the final segmentation map, a segmentation head can be constructed.
  • the input of the segmentation head can be S o and the multi-scale visual features Then get the following output:
  • ⁇ is a two-layer convolutional network
  • each layer is a 3 ⁇ 3 convolution plus ReLU and batch normalization
  • Up represents bilinear interpolation upsampling
  • ⁇ represents 1*1 convolution
  • Feature projection is performed on each pixel of the segmentation head, and the output of the segmentation head is recorded as Y 1 .
  • each second embedding vector corresponds to an object in the image; each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature.
  • the neural network may process the plurality of first embedding vectors into a plurality of second embedding vectors according to the first image feature, wherein each second embedding vector may correspond to a candidate region of the target object, and different second embedding vectors may correspond to different or overlapping candidate regions of the target object.
  • This is equivalent to changing the image features from pixel granularity to target object granularity, that is, introducing target integrity constraints in cross-modal feature fusion, fusing pixels belonging to the same target as a whole with language encoding, and activating instance areas based on targets.
  • This can effectively solve the problems of inaccurate target positioning and mask prediction or detection box prediction in existing language-driven precise instance segmentation methods, thereby improving segmentation accuracy.
  • the neural grid includes multiple transformer layers.
  • the Sentence-object alignment module first generates possible target masks based on the word-pixel aligned features, and then aligns the natural language sentence features and the target masks to more accurately locate the target instance, as shown in Figure 9.
  • the embodiment of the present application designs a mask generator MaskGenerator, which predicts N possible target masks based on the output So of the encoder.
  • a weight can be assigned to each second embedding vector according to the text features, and the second embedding vector with a higher weight is most likely to contain the target object referred to by the text.
  • a weight Q w can be assigned to each mask vector (second embedding vector) of Q o according to the text feature L g .
  • a higher weight in Q w indicates that the corresponding mask is most likely to contain the object referred to by the language.
  • N mask predictions Y N are obtained by multiplying Q o and Y 1.
  • Y N and Q w are multiplied to obtain the final mask prediction M.
  • sim(,) represents the cosine similarity function
  • FIG7 is a schematic diagram of a network architecture of an embodiment of the present application, wherein the overall architecture design follows the classic encoder-decoder paradigm.
  • the encoder part consists of a visual encoder and a language encoder to extract visual and language features, and the Word-Pixel Alignment (WPA) module is in the middle layer of the visual and language encoding to achieve cross-modal interaction.
  • WPA Word-Pixel Alignment
  • a cross-attention layer is used to cross-modally fuse the outputs of the visual and language encoders.
  • the decoder part consists of a mask generator that generates N mask query vectors, a segmentation head that upsamples pixel features, and a Sentence-Object Alignment module (SOA), which weights the output mask query vector according to the sentence features, and uses the weights to perform weighted summation on the segmentation features generated by the segmentation head to obtain the final segmentation mask.
  • SOA Sentence-Object Alignment module
  • each pixel in the image must be classified as foreground or background, so this task can be regarded as a pixel-level binary classification task.
  • this task can be regarded as a pixel-level binary classification task.
  • M′ the values of each point i are m′ i and
  • the segmentation loss is as follows:
  • ⁇ represents the sigmoid function and j represents the jth image in the training batch.
  • a pixel-level contrast loss function can be used as an auxiliary function of the segmentation loss. This function reduces the distance between pixel features within the target object and increases the distance between pixel features within the target object and pixel features outside the object, as shown in Figure 10.
  • an image and a natural language sentence describing an instance target in the image are input.
  • the model will directly predict the mask M or detection box of the instance target, upsample and interpolate it back to the original image size, and binarize it to segment the instance target.
  • RefCOCO RefCOCO
  • RefCOCO+ RefCOCOg
  • G-Ref RefCOCOg
  • the images of these three datasets are all from the MSCOCO dataset, and each is accompanied by different language annotations.
  • the language annotations of RefCOCO and RefCOCO+ are generated through a game called ReferitGame.
  • RefCOCO consists of 142,209 natural language annotations and 19,994 images
  • RefCOCO+ consists of 141,564 natural language annotations and 19,992 images.
  • the main difference between RefCOCO and RefCOCO+ is that RefCOCO+ does not allow the use of positioning words such as "left" and "front” in the language annotations.
  • the RefCOCO+ dataset is more challenging.
  • the language annotations on the G-Ref dataset are from Amazon Mechanical Truk, which contains 85,474 language annotations and 26,711 images.
  • this dataset has two division methods, namely UMD division and Google division.
  • UMD division and Google division the language annotations of G-Ref are more complex and varied, and the average length of its sentences is also greater than the average length of sentences in the RefCOCO and RefCOCO+ datasets, which makes G-Ref a more challenging dataset.
  • the original input data is an RGB image, a 0-1 mask matrix, and a language annotation string.
  • the preprocessing of the image part of the data is as follows: for the training data, the RGB image is normalized, and after regularization, it is scaled to a uniform resolution of 448*448 through bilinear interpolation. At the same time, the 0-1 mask matrix is scaled to the same resolution as the RGB image through nearest neighbor interpolation. For the test data, only the above processing is required for the RGB image, and there is no need to perform nearest neighbor interpolation on the 0-1 mask matrix.
  • the preprocessing of the language part of the data is as follows: Use BertTokenizer in the HuggingFace library to tokenize the input string.
  • BertTokenizer is based on the WordPiece embedding method, and its dictionary size is 30,000. For each tokenized sequence, the first token is a special [CLS] token. If multiple sentences are input, another special [SEP] token will be inserted between sentences.
  • intersection over union is mainly used to indicate the similarity between the predicted area and the true area.
  • intersection-over-union ratio is defined as follows:
  • the global IoU is the sum of the intersections of all test images divided by the sum of their unions.
  • the average IoU is the average of the IoUs of all test images.
  • prec@X is the percentage of images with an IoU greater than a certain threshold X among all test images. In the experiment, the value of X is usually 0.5, 0.6, 0.7, 0.8, or 0.9.
  • CoupAlign is compared with previous SOTA methods on RefCOCO and G-Ref in terms of oIoU.
  • the language annotations provided by the RefCOCO dataset contain many positional words, such as: "The closest girl on the right". This requires the model to not only understand the correspondence between nouns and objects, but also understand the positional relationship between objects represented by positional words.
  • CoupAlign improves 1.97%, 1.94%, and 1.79% on val, testA, and testB of RefCOCO, respectively.
  • the language annotations on G-Ref have more complex grammatical structures than RefCOCO, and the average sentence length is also longer.
  • Word pixel alignment enables cross-modal interaction to occur at both the bottom and high levels of encoding.
  • Table 2 it can be found that after removing the word pixel alignment module, the model's oIoU index dropped by about 4.3%, which shows that the existence of the word pixel alignment module in the encoding stage is very necessary.
  • the model's oIoU index dropped by about 2%. This shows that not only attention from language to vision, but also attention from vision to language is very important.
  • the model's oIoU index dropped by about 1.7%.
  • the mask prediction is visualized in order from large to small similarity to the sentence semantics.
  • the greater the similarity the greater the overlap between the mask and the target object, and the smaller the similarity, the smaller the overlap between the mask and the target object.
  • Figure 11A it can be seen that the sentence target alignment module allows the model to pay attention to different objects, thereby perceiving the positional relationship between objects. And because the target integrity constraint is introduced, the segmentation prediction of the model is less likely to produce hollowing, fragmentation and other phenomena.
  • the present application provides a data processing method, the method comprising: obtaining a first image feature corresponding to an image and a text feature corresponding to a text.
  • the corresponding text features; the semantics of the text corresponds to the target object, and the text indicates the area corresponding to the target object predicted from the image; according to the preset multiple first embedding vectors and the first image features, multiple second embedding vectors are obtained through a neural network, each second embedding vector corresponds to an object in the image; each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature; according to the similarity between the text features and the multiple second embedding vectors, the weight corresponding to each second embedding vector is determined, and the multiple weights are used to fuse with the multiple second image features to determine the predicted area corresponding to the target object.
  • the present application also provides a data processing method, the method comprising:
  • each second embedding vector corresponds to an object in the image
  • each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature
  • the feature extraction network and the neural network are updated according to the difference between the predicted area and the real area corresponding to the target object in the image.
  • the predicted area is a mask area or a detection box.
  • the semantics of the text corresponds to the target object, specifically including: the semantics of the text is used to describe the characteristics of the target object.
  • acquiring a first image feature corresponding to the image and a text feature corresponding to the text includes:
  • the third image feature and the first text feature are fused through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • FIG. 12 is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application.
  • a data processing device 1200 provided in an embodiment of the present application includes:
  • the processing module 1201 is used to obtain a first image feature corresponding to an image and a text feature corresponding to a text; the semantics of the text corresponds to a target object, and the text indicates a region corresponding to the target object predicted from the image;
  • each second embedding vector corresponds to an object in the image
  • each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature
  • a weight corresponding to each of the second embedding vectors is determined, and the multiple weights are used to be fused with the multiple second image features to determine the predicted area corresponding to the target object.
  • processing module 1201 can refer to the description of steps 601 to 603 in the above embodiment, which will not be repeated here.
  • the image may include multiple objects including the target object, each second embedding vector corresponds to an object in the image, and one or more embedding vectors in the multiple second embedding vectors may correspond to the target object.
  • the "correspondence" here can be understood as the second embedding vector is used to describe the characteristics of an object in the image, and the second embedding vector obtained by the neural network can distinguish different objects in the image, so that the subsequent prediction process can be based on the object granularity.
  • This is equivalent to changing the image features from pixel granularity to target object granularity, that is, introducing target integrity constraints in cross-modal feature fusion, fusing pixels belonging to the same target as a whole with language encoding, and activating instance regions based on targets.
  • This can effectively solve the problems of inaccurate target positioning and mask prediction or detection box prediction in existing language-driven precise instance segmentation methods, thereby Improve the processing accuracy of the model.
  • the predicted area is a mask area or a detection box.
  • the semantics of the text corresponds to the target object, specifically including: the semantics of the text is used to describe the characteristics of the target object.
  • the processing module is specifically configured to:
  • the third image feature and the first text feature are fused through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • the first image feature is a feature that is upsampled to a size consistent with the image.
  • the neural grid includes multiple transformer layers.
  • the present application also provides a data processing device, including:
  • a processing module used for obtaining a first image feature corresponding to an image and a text feature corresponding to a text; the semantics of the text corresponds to a target object, and the text indicates a region corresponding to the target object predicted from the image; the first image feature and the text feature are obtained according to a feature extraction network;
  • each second embedding vector corresponds to an object in the image
  • each second embedding vector and the first image feature are used to fuse to obtain a corresponding second image feature
  • An updating module is used to update the feature extraction network and the neural network according to the difference between the predicted area and the real area corresponding to the target object in the image.
  • the image may include multiple objects including the target object, each second embedding vector corresponds to an object in the image, and one or more embedding vectors in the multiple second embedding vectors may correspond to the target object.
  • the "correspondence" here can be understood as the second embedding vector is used to describe the characteristics of an object in the image, and the second embedding vector obtained by the neural network can distinguish different objects in the image, so that the subsequent prediction process can be based on the object granularity.
  • This is equivalent to changing the image features from pixel granularity to target object granularity, that is, introducing target integrity constraints in cross-modal feature fusion, fusing pixels belonging to the same target as a whole with language encoding, and activating instance areas based on targets.
  • This can effectively solve the problems of inaccurate target positioning and mask prediction or detection box prediction in existing language-driven precise instance segmentation methods, thereby improving the processing accuracy of the model.
  • the predicted area is a mask area or a detection box.
  • the semantics of the text corresponds to the target object, specifically including: the semantics of the text is used to describe the characteristics of the target object.
  • the processing module is specifically configured to:
  • the third image feature and the first text feature are fused through a bidirectional attention mechanism to obtain the first image feature corresponding to the image and the text feature corresponding to the text.
  • FIG. 13 is a structural schematic diagram of an execution device provided in an embodiment of the present application.
  • the execution device 1300 can be specifically manifested as a virtual reality VR device, a mobile phone, a tablet, a laptop, a smart wearable device, a monitoring data processing device or a server, etc., which is not limited here.
  • the execution device 1300 includes: a receiver 1301, a transmitter 1302, a processor 1303 and a memory 1304 (wherein the number of processors 1303 in the execution device 1300 can be one or more, and one processor is taken as an example in Figure 13), wherein the processor 1303 may include an application processor 13031 and a communication processor 13032.
  • the receiver 1301, the transmitter 1302, the processor 1303 and the memory 1304 may be connected via a bus or other means.
  • the memory 1304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1303. A portion of the memory 1304 may also include a non-volatile random access memory (NVRAM).
  • NVRAM non-volatile random access memory
  • the memory 1304 stores processors and operation instructions, executable modules or data structures, or a subset thereof, or an extended set thereof, wherein the operation instructions may include various operation instructions for implementing various operations.
  • the processor 1303 controls the operation of the execution device.
  • the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc.
  • the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc.
  • various buses are referred to as bus systems in the figure.
  • the method disclosed in the above embodiment of the present application can be applied to the processor 1303, or implemented by the processor 1303.
  • the processor 1303 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1303 or the instruction in the form of software.
  • the above processor 1303 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
  • the processor 1303 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present application.
  • the general processor can be a microprocessor or the processor can also be any conventional processor, etc.
  • the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to be executed, or a combination of hardware and software modules in the decoding processor can be executed.
  • the software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc.
  • the storage medium is located in the memory 1304, and the processor 1303 reads the information in the memory 1304 and completes the steps of the above method involving the model reasoning process in combination with its hardware.
  • the receiver 1301 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device.
  • the transmitter 1302 can be used to output digital or character information through the first interface; the transmitter 1302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1302 can also include a display device such as a display screen.
  • FIG. 14 is a structural diagram of a training device provided by the embodiment of the present application.
  • the training device 1400 is implemented by one or more servers.
  • the training device 1400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1414 (for example, one or more processors) and a memory 1432, and one or more storage media 1430 (for example, one or more mass storage devices) storing application programs 1442 or data 1444.
  • the memory 1432 and the storage medium 1430 can be short-term storage or permanent storage.
  • the program stored in the storage medium 1430 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processor 1414 can be configured to communicate with the storage medium 1430 to execute a series of instruction operations in the storage medium 1430 on the training device 1400.
  • the training device 1400 may also include one or more power supplies 1426, one or more wired or wireless network interfaces 1450, one or more input and output interfaces 1458; or, one or more operating systems 1441, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.
  • the central processing unit 1414 is used to execute actions related to model training in the above embodiments.
  • Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
  • a computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing.
  • the computer-readable storage medium When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.
  • the execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input/output interface, a pin or a circuit, etc.
  • the processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment.
  • the storage unit is a storage unit in the chip, such as a register, a cache, etc.
  • the storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
  • ROM read-only memory
  • RAM random access memory
  • FIG. 15 is a schematic diagram of a structure of a chip provided in an embodiment of the present application.
  • the chip can be expressed as a neural network processor NPU 1500.
  • NPU 1500 is mounted on the host CPU (Host CPU) as a coprocessor, and tasks are assigned by the Host CPU.
  • the core part of the NPU is the operation circuit 1503, which is controlled by the controller 1504 to extract matrix data from the memory and perform multiplication operations.
  • the operation circuit 1503 includes multiple processing units (Process Engine, PE) inside.
  • the operation circuit 1503 is a two-dimensional systolic array.
  • the operation circuit 1503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition.
  • the operation circuit 1503 is a general-purpose matrix processor.
  • the operation circuit takes the corresponding data of matrix B from the weight memory 1502 and caches it on each PE in the operation circuit.
  • the operation circuit takes the matrix A data from the input memory 1501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is stored in the accumulator 1508.
  • Unified memory 1506 is used to store input data and output data. Weight data is directly transferred to weight memory 1502 through Direct Memory Access Controller (DMAC) 1505. Input data is also transferred to unified memory 1506 through DMAC.
  • DMAC Direct Memory Access Controller
  • BIU stands for Bus Interface Unit, that is, the bus interface unit 1510, which is used for the interaction between AXI bus and DMAC and instruction fetch buffer (IFB) 1509.
  • IOB instruction fetch buffer
  • the bus interface unit 1510 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
  • BIU Bus Interface Unit
  • DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or to transfer weight data to the weight memory 1502 or to transfer input data to the input memory 1501.
  • the vector calculation unit 1507 includes multiple operation processing units, which further process the output of the operation circuit 1503 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional/fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.
  • the vector calculation unit 1507 can store the processed output vector to the unified memory 1506.
  • the vector calculation unit 1507 can apply a linear function; or a nonlinear function to the output of the operation circuit 1503, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values to generate an activation value.
  • the vector calculation unit 1507 generates a normalized value, a pixel-level summed value, or both.
  • the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in a subsequent layer in a neural network.
  • An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;
  • Unified memory 1506, input memory 1501, weight memory 1502 and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.
  • the processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above programs.
  • the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.
  • the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
  • the technical solution of the present application can be essentially or in other words, the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or CD, etc., including a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application. Law.
  • a readable storage medium such as a computer floppy disk, U disk, mobile hard disk, ROM, RAM, disk or CD, etc.
  • all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof.
  • all or part of the embodiments may be implemented in the form of a computer program product.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.
  • the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center.
  • the computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations.
  • the available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
  • a magnetic medium e.g., a floppy disk, a hard disk, a tape
  • an optical medium e.g., a DVD
  • a semiconductor medium e.g., a solid-state drive (SSD)

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Medical Informatics (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Image Analysis (AREA)

Abstract

一种数据处理方法，应用于图像处理，例如图像分割或者目标检测，方法包括：获取图像对应的第一图像特征以及文本对应的文本特征；根据预设的多个第一嵌入向量以及第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于目标对象的一个候选区域；每个第二嵌入向量和第一图像特征用于融合得到一个对应的第二图像特征；根据文本特征和多个第二嵌入向量之间的相似度，确定每个第二嵌入向量对应的权重，多个权重用于和多个第二图像特征进行加权，以确定目标对象对应的预测区域。本申请将图像特征从像素粒度变为以目标对象为粒度，将属于同个目标的像素当做一个整体来和语言编码进行融合，可以提升模型的处理精度。

Description

一种数据处理方法及其装置
本申请要求于2022年10月20日提交中国国家知识产权局、申请号为202211292146.9、发明名称为“一种数据处理方法及其装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人工智能领域，尤其涉及一种数据处理方法及其装置。
背景技术
人工智能(Artificial Intelligence，AI)是利用数字计算机或者数字计算机控制的机器模拟、延伸和扩展人的智能，感知环境、获取知识并使用知识获得最佳结果的理论、方法、技术及应用系统。换句话说，人工智能是计算机科学的一个分支，它企图了解智能的实质，并生产出一种新的能以人类智能相似的方式作出反应的智能机器。人工智能也就是研究各种智能机器的设计原理与实现方法，使机器具有感知、推理与决策的功能。
语言驱动的精确实例分割，是一种特殊的语义分割技术，指根据自然语言指导，在一幅图片中精确地分割出语言所描述的实例目标，其特点在于：1)传统语义分割模型为为属于同个类别的所有目标都预测同样的标签，并不区分同个类别中不同的目标，而语言驱动的精确实例分割，需要从多个同类目标中精确地识别出语言描述所对应的一个实例目标；2)语义分割模型需要预先定义一组语义类别的标签，从而学习分割出这些类别的目标，而语言驱动的精确实例分割可以接收更加灵活的自然语言输入，且不限目标类别。
由于自然语言输入的灵活性，语言驱动的实例分割方法主要依靠将自然语言句子编码和图像的视觉编码进行融合，从而在视觉特征图上激活和语言编码相关性高的区域，然而这种跨模态特征融合的方案其挑战主要在于两个方面，首先是实例目标定位不准确，在多个拥挤的同类目标中，无法准确地锁定单个实例目标；其次是预测的掩膜不够准确，容易溢出到相邻的同类目标上，上述问题也同样存在于目标检测任务中。
发明内容
本申请提供了一种数据处理方法，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而提升模型的处理精度。
第一方面，本申请提供了一种数据处理方法，包括：获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合(例如，加权)，以确定所述目标对象对应的预测区域。
其中，图像可以包括目标对象在内的多个对象，每个第二嵌入向量对应于所述图像中的一个对象，多个第二嵌入向量中的一个或多个嵌入向量可以对应于目标对象。应理解，这里的“对应”可以理解为，第二嵌入向量用于描述所述图像中的一个对象的特征，通过神经网络得到的第二嵌入向量可以将图像中的不同对象进行区分，以便后续的预测过程中能够以对象为粒度。
相当于将图像特征从像素粒度变为以目标对象为粒度，也就是在跨模态特征融合中引入目标整体性约束，将属于同个目标的像素当做一个整体来和语言编码进行融合，以目标为单位来激活实例区域，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而提升模型的处理精度。
在一种可能的实现中，所述预测区域为掩码区域或者检测框。
在一种可能的实现中，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
在一种可能的实现中，所述获取图像对应的第一图像特征以及文本对应的文本特征，包括：
通过图像编码器处理所述图像，得到所述图像对应的图像特征；
通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
在一种可能的实现中，所述第一图像特征为上采样到和所述图像的尺寸一致的特征。
在一种可能的实现中，所述神经网格包括多个transformer层。
第二方面，一种数据处理方法，包括：
获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；所述第一图像特征以及所述文本特征为根据特征提取网络得到的；
根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域；
根据所述预测区域和所述图像中所述目标对象对应的真实区域之间的差异，更新所述特征提取网络以及所述神经网络。
其中，图像可以包括目标对象在内的多个对象，每个第二嵌入向量对应于所述图像中的一个对象，多个第二嵌入向量中的一个或多个嵌入向量可以对应于目标对象。应理解，这里的“对应”可以理解为，第二嵌入向量用于描述所述图像中的一个对象的特征，通过神经网络得到的第二嵌入向量可以将图像中的不同对象进行区分，以便后续的预测过程中能够以对象为粒度。
相当于将图像特征从像素粒度变为以目标对象为粒度，也就是在跨模态特征融合中引入目标整体性约束，将属于同个目标的像素当做一个整体来和语言编码进行融合，以目标为单位来激活实例区域，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而提升模型的处理精度。
在一种可能的实现中，所述预测区域为掩码区域或者检测框。
在一种可能的实现中，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
在一种可能的实现中，所述获取图像对应的第一图像特征以及文本对应的文本特征，包括：
通过图像编码器处理所述图像，得到所述图像对应的图像特征；
通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
第三方面，本申请提供了一种数据处理装置，包括：
处理模块，用于获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；
根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每 个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域。
其中，图像可以包括目标对象在内的多个对象，每个第二嵌入向量对应于所述图像中的一个对象，多个第二嵌入向量中的一个或多个嵌入向量可以对应于目标对象。应理解，这里的“对应”可以理解为，第二嵌入向量用于描述所述图像中的一个对象的特征，通过神经网络得到的第二嵌入向量可以将图像中的不同对象进行区分，以便后续的预测过程中能够以对象为粒度。
相当于将图像特征从像素粒度变为以目标对象为粒度，也就是在跨模态特征融合中引入目标整体性约束，将属于同个目标的像素当做一个整体来和语言编码进行融合，以目标为单位来激活实例区域，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而提升模型的处理精度。
在一种可能的实现中，所述预测区域为掩码区域或者检测框。
在一种可能的实现中，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
在一种可能的实现中，所述处理模块，具体用于：
通过图像编码器处理所述图像，得到所述图像对应的图像特征；
通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
在一种可能的实现中，所述第一图像特征为上采样到和所述图像的尺寸一致的特征。
在一种可能的实现中，所述神经网格包括多个transformer层。
第四方面，本申请提供了一种数据处理装置，包括：
处理模块，用于获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；所述第一图像特征以及所述文本特征为根据特征提取网络得到的；
根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域；
更新模块，用于根据所述预测区域和所述图像中所述目标对象对应的真实区域之间的差异，更新所述特征提取网络以及所述神经网络。
在一种可能的实现中，所述预测区域为掩码区域或者检测框。
在一种可能的实现中，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
在一种可能的实现中，所述处理模块，具体用于：
通过图像编码器处理所述图像，得到所述图像对应的图像特征；
通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
第五方面，本申请实施例提供了一种数据处理装置，可以包括存储器、处理器以及总线系统，其中，存储器用于存储程序，处理器用于执行存储器中的程序，以执行如上述第一方面及其任一可选的方法、以及如上述第二方面及其任一可选的方法。
第六方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质中存储有计算机程序，当其在计算机上运行时，使得计算机执行上述第一方面及其任一可选的方法、以及上述第二方面及其任一可选的方法。
第七方面，本申请实施例提供了一种计算机程序，当其在计算机上运行时，使得计算机执行上述第一方面及其任一可选的方法、以及上述第二方面及其任一可选的方法。
第八方面，本申请提供了一种芯片系统，该芯片系统包括处理器，用于支持执行数据处理装置实现上述方面中所涉及的功能，例如，发送或处理上述方法中所涉及的数据；或，信息。在一种可能的设计中，所述芯片系统还包括存储器，所述存储器，用于保存执行设备或训练设备必要的程序指令和数据。该芯片系统，可以由芯片构成，也可以包括芯片和其他分立器件。
附图说明
图1A为人工智能主体框架的一种结构示意图；
图1B和至图1C为本申请的应用系统框架示意；
图1D为终端的一种可选的硬件结构示意图；
图2为一种服务器的结构示意图；
图3为本申请的一种系统架构示意；
图4为一种云服务的流程；
图5为一种网络的结构示意；
图6为本申请实施例提供的一种数据处理方法的流程示意；
图7至图10为本申请实施例提供的一种数据处理方法的流程示意；
图11A和图11B为本申请的一种有益效果示意；
图12为本申请实施例提供的数据处理装置的一种结构示意图；
图13为本申请实施例提供的执行设备的一种结构示意图；
图14为本申请实施例提供的训练设备一种结构示意图；
图15为本申请实施例提供的芯片的一种结构示意图。
具体实施方式
下面结合本申请实施例中的附图对本申请实施例进行描述。本申请的实施方式部分使用的术语仅用于对本申请的具体实施例进行解释，而非旨在限定本申请。
下面结合附图，对本申请的实施例进行描述。本领域普通技术人员可知，随着技术的发展和新场景的出现，本申请实施例提供的技术方案对于类似的技术问题，同样适用。
本申请的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，这仅仅是描述本申请的实施例中对相同属性的对象在描述时所采用的区分方式。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，以便包含一系列单元的过程、方法、系统、产品或设备不必限于那些单元，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它单元。
本文中所用用语“基本(substantially)”、“大约(about)”及类似用语用作近似用语、而并非用作程度用语，且旨在考虑到所属领域中的普通技术人员将知的测量值或计算值的固有偏差。此外，在阐述本申 请实施例时使用“可(may)”是指“可能的一个或多个实施例”。本文中所用用语“使用(use)”、“正使用(using)”、及“被使用(used)”可被视为分别与用语“利用(utilize)”、“正利用(utilizing)”、及“被利用(utilized)”同义。另外，用语“示例性(exemplary)”旨在指代实例或例示。
首先对人工智能系统总体工作流程进行描述，请参见图1A，图1A示出的为人工智能主体框架的一种结构示意图，下面从“智能信息链”(水平轴)和“IT价值链”(垂直轴)两个维度对上述人工智能主题框架进行阐述。其中，“智能信息链”反映从数据的获取到处理的一列过程。举例来说，可以是智能信息感知、智能信息表示与形成、智能推理、智能决策、智能执行与输出的一般过程。在这个过程中，数据经历了“数据—信息—知识—智慧”的凝练过程。“IT价值链”从人智能的底层基础设施、信息(提供和处理技术实现)到系统的产业生态过程，反映人工智能为信息技术产业带来的价值。
(1)基础设施
基础设施为人工智能系统提供计算能力支持，实现与外部世界的沟通，并通过基础平台实现支撑。通过传感器与外部沟通；计算能力由智能芯片(CPU、NPU、GPU、ASIC、FPGA等硬件加速芯片)提供；基础平台包括分布式计算框架及网络等相关的平台保障和支持，可以包括云存储和计算、互联互通网络等。举例来说，传感器和外部沟通获取数据，这些数据提供给基础平台提供的分布式计算系统中的智能芯片进行计算。
(2)数据
基础设施的上一层的数据用于表示人工智能领域的数据来源。数据涉及到图形、图像、语音、文本，还涉及到传统设备的物联网数据，包括已有系统的业务数据以及力、位移、液位、温度、湿度等感知数据。
(3)数据处理
数据处理通常包括数据训练，机器学习，深度学习，搜索，推理，决策等方式。
其中，机器学习和深度学习可以对数据进行符号化和形式化的智能信息建模、抽取、预处理、训练等。
推理是指在计算机或智能系统中，模拟人类的智能推理方式，依据推理控制策略，利用形式化的信息进行机器思维和求解问题的过程，典型的功能是搜索与匹配。
决策是指智能信息经过推理后进行决策的过程，通常提供分类、排序、预测等功能。
(4)通用能力
对数据经过上面提到的数据处理后，进一步基于数据处理的结果可以形成一些通用的能力，比如可以是算法或者一个通用系统，例如，翻译，文本的分析，计算机视觉的处理，语音识别，图像的识别等等。
(5)智能产品及行业应用
智能产品及行业应用指人工智能系统在各领域的产品和应用，是对人工智能整体解决方案的封装，将智能信息决策产品化、实现落地应用，其应用领域主要包括：智能终端、智能交通、智能医疗、自动驾驶、智慧城市等。
本申请可以应用于人工智能领域的图像处理领域中，下面以图像处理为例将对多个落地到产品的多个应用场景进行介绍。
首先介绍本申请的应用场景。
在一些场景中，图像处理功能可以作为机器人视觉语言导航系统的一个核心算法模块。例如，用户通过语言指令，想命令一个家用机器人走到椅子旁边，拿一个花瓶。而机器人需要精确地将语言描述的目标实例椅子和花瓶都分割或者检测出来，然后才能完成拿花瓶的任务。
在一些场景中，图像处理功能可以应用于自动驾驶平台中，当用户和智能驾驶系统进行自然语言交互时，用户要求停靠在右前方的黄色汽车之后，那么智能驾驶系统的识别模块需要首先理解用户的自然语言指令，并且精确地分割或者检测出黄色车，从而完成用户的要求。
在一些场景中，图像处理功能可以应用于交互式图像编辑系统中，该系统需要根据用户自然语言描述的需求来修改图片，图像处理功能可以定位用户想要修改的区域，再结合现有的图片编辑工具来修改图片内容。
本申请可以但不限于应用在图像处理功能的应用程序(以下可以简称为图像处理类应用程序)或者云侧服务器提供的云服务等，接下来分别进行介绍：
一、语言驱动的图像处理类应用程序
本申请实施例的产品形态可以为图像处理类应用程序，特别的，可以为语言驱动的图像处理类应用程序。语言驱动的图像处理类应用程序可以运行在终端设备或者云侧的服务器上。
在一种可能的实现中，语言驱动的图像处理类应用程序可以实现基于输入的图像以及文本进行图像分割或者是目标检测等任务，得到处理结果，处理结果可以是图像分割结果(掩码区域)以及检测框，图像分割结果(掩码区域)以及检测框内可以包含文本的语义所指示的对象(例如本申请实施例中的目标对象)。
在一种可能的实现中，用户可以打开终端设备上安装的图像处理类应用程序，并输入图像以及文本，图像处理类应用程序可以通过本申请实施例提供的方法对图像和文本进行处理，并将处理结果呈现给用户(呈现方式可以但不限于是显示、保存、上传到云侧等)。
在一种可能的实现中，用户可以打开终端设备上安装的图像处理类应用程序，并输入图像，图像处理类应用程序可以将图像发送至云侧的服务器，云侧的服务器通过本申请实施例提供的方法对图像和文本进行处理，并将处理结果回传至终端设备，终端设备可以将处理结果呈现给用户(呈现方式可以但不限于是显示、保存、上传到云侧等)。
接下来分别从功能架构以及实现功能的产品架构介绍本申请实施例中的图像处理类应用程序。
参照图1B，图1B为本申请实施例中图像处理类应用程序的功能架构示意：
在一种可能的实现中，如图1B所示，图像处理类应用程序102可接收输入的参数101(例如包含图像)且产生处理结果103。图像处理类应用程序102可在(举例来说)至少一个计算机系统上执行，且包括计算机代码，所述计算机代码在由一或多个计算机执行时致使所述计算机执行用于执行本申请实施例提供的方法。
参照图1C，图1C为本申请实施例中运行图像处理类应用程序的实体架构示意：
参见图1C，图1C示出了一种系统架构示意图。该系统可以包括终端100、以及服务器200。其中，服务器200可以包括一个或者多个服务器(图1C中以包括一个服务器作为示例进行说明)，服务器200可以为一个或者多个终端提供本申请实施例提供的方法。
其中，终端100上可以安装有图像处理类应用程序，上述应用程序和网页可以提供一个界面，终端100可以接收用户在语言驱动的图像处理界面上输入的相关参数，并将上述参数发送至服务器200，服务器200可以基于接收到的参数，得到处理结果，并将处理结果返回至至终端100。
应理解，在一些可选的实现中，终端100也可以由自身完成基于接收到的参数，得到处理结果的动作，而不需要服务器配合实现，本申请实施例并不限定。
接下来描述图1C中终端100的产品形态；
本申请实施例中的终端100可以为手机、平板电脑、可穿戴设备、车载设备、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)等，本申请实施例对此不作任何限制。
图1D示出了终端100的一种可选的硬件结构示意图。
参考图1D所示，终端100可以包括射频单元110、存储器120、输入单元130、显示单元140、摄像头150(可选的)、音频电路160(可选的)、扬声器161(可选的)、麦克风162(可选的)、处理器170、外部接口180、电源190等部件。本领域技术人员可以理解，图1D仅仅是终端或多功能设备的举例，并不构成对终端或多功能设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件。
输入单元130可用于接收输入的数字或字符信息，以及产生与该便携式多功能装置的用户设置以及功能控制有关的键信号输入。具体地，输入单元130可包括触摸屏131(可选的)和/或其他输入设备132。该触摸屏131可收集用户在其上或附近的触摸操作(比如用户使用手指、关节、触笔等任何适合的物体在触摸屏上或在触摸屏附近的操作)，并根据预先设定的程序驱动相应的连接装置。触摸屏可以检测用户 对触摸屏的触摸动作，将该触摸动作转换为触摸信号发送给该处理器170，并能接收该处理器170发来的命令并加以执行；该触摸信号至少包括触点坐标信息。该触摸屏131可以提供该终端100和用户之间的输入界面和输出界面。此外，可以采用电阻式、电容式、红外线以及表面声波等多种类型实现触摸屏。除了触摸屏131，输入单元130还可以包括其他输入设备。具体地，其他输入设备132可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆等中的一种或多种。
其中，输入设备132可以接收到输入的图像、文本等等。
该显示单元140可用于显示由用户输入的信息或提供给用户的信息、终端100的各种菜单、交互界面、文件显示和/或任意一种多媒体文件的播放。在本申请实施例中，显示单元140可用于显示图像处理类应用程序的界面、处理结果等。
该存储器120可用于存储指令和数据，存储器120可主要包括存储指令区和存储数据区，存储数据区可存储各种数据，如多媒体文件、文本等；存储指令区可存储操作系统、应用、至少一个功能所需的指令等软件单元，或者他们的子集、扩展集。还可以包括非易失性随机存储器；向处理器170提供包括管理计算处理设备中的硬件、软件以及数据资源，支持控制软件和应用。还用于多媒体文件的存储，以及运行程序和应用的存储。
处理器170是终端100的控制中心，利用各种接口和线路连接整个终端100的各个部分，通过运行或执行存储在存储器120内的指令以及调用存储在存储器120内的数据，执行终端100的各种功能和处理数据，从而对终端设备进行整体控制。可选的，处理器170可包括一个或多个处理单元；优选的，处理器170可集成应用处理器和调制解调处理器，其中，应用处理器主要处理操作系统、用户界面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器170中。在一些实施例中，处理器、存储器、可以在单一芯片上实现，在一些实施例中，他们也可以在独立的芯片上分别实现。处理器170还可以用于产生相应的操作控制信号，发给计算处理设备相应的部件，读取以及处理软件中的数据，尤其是读取和处理存储器120中的数据和程序，以使其中的各个功能模块执行相应的功能，从而控制相应的部件按指令的要求进行动作。
其中，存储器120可以用于存储数据处理方法相关的软件代码，处理器170可以执行芯片的数据处理方法的步骤，也可以调度其他单元(例如上述输入单元130以及显示单元140)以实现相应的功能。
该射频单元110(可选的)可用于收发信息或通话过程中信号的接收和发送，例如，将基站的下行信息接收后，给处理器170处理；另外，将设计上行的数据发送给基站。通常，RF电路包括但不限于天线、至少一个放大器、收发信机、耦合器、低噪声放大器(Low Noise Amplifier，LNA)、双工器等。此外，射频单元110还可以通过无线通信与网络设备和其他设备通信。该无线通信可以使用任一通信标准或协议，包括但不限于全球移动通讯系统(Global System of Mobile communication，GSM)、通用分组无线服务(General Packet Radio Service，GPRS)、码分多址(Code Division Multiple Access，CDMA)、宽带码分多址(Wideband Code Division Multiple Access，WCDMA)、长期演进(Long Term Evolution，LTE)、电子邮件、短消息服务(Short Messaging Service，SMS)等。
其中，在本申请实施例中，该射频单元110可以将图像发送至服务器200，并接收到服务器200发送的处理结果。
应理解，该射频单元110为可选的，其可以被替换为其他通信接口，例如可以是网口。
终端100还包括给各个部件供电的电源190(比如电池)，优选的，电源可以通过电源管理系统与处理器170逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。
终端100还包括外部接口180，该外部接口可以是标准的Micro USB接口，也可以使多针连接器，可以用于连接终端100与其他装置进行通信，也可以用于连接充电器为终端100充电。
尽管未示出，终端100还可以包括闪光灯、无线保真(wireless fidelity，WiFi)模块、蓝牙模块、不同功能的传感器等，在此不再赘述。下文中描述的部分或全部方法均可以应用在如图1D所示的终端100中。
接下来描述图1C中服务器200的产品形态；
图2提供了一种服务器200的结构示意图，如图2所示，服务器200包括总线201、处理器202、通信接口203和存储器204。处理器202、存储器204和通信接口203之间通过总线201通信。
总线201可以是外设部件互连标准(peripheral component interconnect，PCI)总线或扩展工业标准结 构(extended industry standard architecture，EISA)总线等。总线可以分为地址总线、数据总线、控制总线等。为便于表示，图2中仅用一条粗线表示，但并不表示仅有一根总线或一种类型的总线。
处理器202可以为中央处理器(central processing unit，CPU)、图形处理器(graphics processing unit，GPU)、微处理器(micro processor，MP)或者数字信号处理器(digital signal processor，DSP)等处理器中的任意一种或多种。
存储器204可以包括易失性存储器(volatile memory)，例如随机存取存储器(random access memory，RAM)。存储器204还可以包括非易失性存储器(non-volatile memory)，例如只读存储器(read-only memory，ROM)，快闪存储器，机械硬盘(hard drive drive，HDD)或固态硬盘(solid state drive，SSD)。
其中，存储器204可以用于存储数据处理方法相关的软件代码，处理器202可以执行芯片的数据处理方法的步骤，也可以调度其他单元以实现相应的功能。
应理解，上述终端100和服务器200可以为集中式或者是分布式的设备，上述终端100和服务器200中的处理器(例如处理器170以及处理器202)可以为硬件电路(如专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)、通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器等等)、或这些硬件电路的组合，例如，处理器可以为具有执行指令功能的硬件系统，如CPU、DSP等，或者为不具有执行指令功能的硬件系统，如ASIC、FPGA等，或者为上述不具有执行指令功能的硬件系统以及具有执行指令功能的硬件系统的组合。
应理解，本申请实施例中的和模型推理过程相关的步骤涉及AI相关的运算，在执行AI运算时，终端设备和服务器的指令执行架构不仅仅局限在上述介绍的处理器结合存储器的架构。下面结合图3对本申请实施例提供的系统架构进行详细的介绍。
图3为本申请实施例提供的系统架构示意图。如图3所示，系统架构500包括执行设备510、训练设备520、数据库530、客户设备540、数据存储系统550以及数据采集系统560。
执行设备510包括计算模块511、I/O接口512、预处理模块513和预处理模块514。计算模块511中可以包括目标模型/规则501，预处理模块513和预处理模块514是可选的。
其中，执行设备510可以为上述运行图像处理类应用程序的终端设备或者服务器。
数据采集设备560用于采集训练样本。训练样本可以为多个图像等。在采集到训练样本之后，数据采集设备560将这些训练样本存入数据库530。
训练设备520可以基于数据库530中维护训练样本，对待训练的神经网络(例如本申请实施例中的跨模态语言模型(例如包括文本编码器、图像编码器、目标编码器等))，以得到目标模型/规则501。
应理解，训练设备520可以基于数据库530中维护训练样本，对待训练的神经网络进行预训练过程，或者是在预训练的基础上进行模型的微调。
需要说明的是，在实际应用中，数据库530中维护的训练样本不一定都来自于数据采集设备560的采集，也有可能是从其他设备接收得到的。另外需要说明的是，训练设备520也不一定完全基于数据库530维护的训练样本进行目标模型/规则501的训练，也有可能从云端或其他地方获取训练样本进行模型训练，上述描述不应该作为对本申请实施例的限定。
根据训练设备520训练得到的目标模型/规则501可以应用于不同的系统或设备中，如应用于图3所示的执行设备510，该执行设备510可以是终端，如手机终端，平板电脑，笔记本电脑，增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)设备，车载终端等，还可以是服务器等。
具体的，训练设备520可以将训练后的模型传递至执行设备510。
在图3中，执行设备510配置输入/输出(input/output，I/O)接口512，用于与外部设备进行数据交互，用户可以通过客户设备540向I/O接口512输入数据(例如本申请实施例中的图像等)。
预处理模块513和预处理模块514用于根据I/O接口512接收到的输入数据进行预处理。应理解，可以没有预处理模块513和预处理模块514或者只有的一个预处理模块。当不存在预处理模块513和预处理模块514时，可以直接采用计算模块511对输入数据进行处理。
在执行设备510对输入数据进行预处理，或者在执行设备510的计算模块511执行计算等相关的处理过程中，执行设备510可以调用数据存储系统550中的数据、代码等以用于相应的处理，也可以将相应处 理得到的数据、指令等存入数据存储系统550中。
最后，I/O接口512将处理结果提供给客户设备540，从而提供给用户。
在图3所示情况下，用户可以手动给定输入数据，该“手动给定输入数据”可以通过I/O接口512提供的界面进行操作。另一种情况下，客户设备540可以自动地向I/O接口512发送输入数据，如果要求客户设备540自动发送输入数据需要获得用户的授权，则用户可以在客户设备540中设置相应权限。用户可以在客户设备540查看执行设备510输出的结果，具体的呈现形式可以是显示、声音、动作等具体方式。客户设备540也可以作为数据采集端，采集如图所示输入I/O接口512的输入数据及输出I/O接口512的输出结果作为新的样本数据，并存入数据库530。当然，也可以不经过客户设备540进行采集，而是由I/O接口512直接将如图所示输入I/O接口512的输入数据及输出I/O接口512的输出结果，作为新的样本数据存入数据库530。
值得注意的是，图3仅是本申请实施例提供的一种系统架构的示意图，图中所示设备、器件、模块等之间的位置关系不构成任何限制，例如，在图3中，数据存储系统550相对执行设备510是外部存储器，在其它情况下，也可以将数据存储系统550置于执行设备510中。应理解，上述执行设备510可以部署于客户设备540中。
从模型的推理侧来说：
本申请实施例中，上述执行设备520的计算模块511可以获取到数据存储系统550中存储的代码来实现本申请实施例中的和模型推理过程相关的步骤。
本申请实施例中，执行设备520的计算模块511可以包括硬件电路(如专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)、通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器等等)、或这些硬件电路的组合，例如，训练设备520可以为具有执行指令功能的硬件系统，如CPU、DSP等，或者为不具有执行指令功能的硬件系统，如ASIC、FPGA等，或者为上述不具有执行指令功能的硬件系统以及具有执行指令功能的硬件系统的组合。
具体的，执行设备520的计算模块511可以为具有执行指令功能的硬件系统，本申请实施例提供的和模型推理过程相关的步骤可以为存储在存储器中的软件代码，执行设备520的计算模块511可以从存储器中获取到软件代码，并执行获取到的软件代码来实现本申请实施例提供的和模型推理过程相关的步骤。
应理解，执行设备520的计算模块511可以为不具有执行指令功能的硬件系统以及具有执行指令功能的硬件系统的组合，本申请实施例提供的和模型推理过程相关的步骤的部分步骤还可以通过执行设备520的计算模块511中不具有执行指令功能的硬件系统来实现，这里并不限定。
从模型的训练侧来说：
本申请实施例中，上述训练设备520可以获取到存储器(图3中未示出，可以集成于训练设备520或者与训练设备520分离部署)中存储的代码来实现本申请实施例中和模型训练相关的步骤。
本申请实施例中，训练设备520可以包括硬件电路(如专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)、通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器等等)、或这些硬件电路的组合，例如，训练设备520可以为具有执行指令功能的硬件系统，如CPU、DSP等，或者为不具有执行指令功能的硬件系统，如ASIC、FPGA等，或者为上述不具有执行指令功能的硬件系统以及具有执行指令功能的硬件系统的组合。
应理解，训练设备520可以为不具有执行指令功能的硬件系统以及具有执行指令功能的硬件系统的组合，本申请实施例提供的中和模型训练相关的部分步骤还可以通过训练设备520中不具有执行指令功能的硬件系统来实现，这里并不限定。
二、服务器提供的语言驱动的图像处理类云服务：
在一种可能的实现中，服务器可以通过应用程序编程接口(application programming interface，API)为端侧提供语言驱动的图像处理的服务。
其中，终端设备可以通过云端提供的API，将相关参数(例如图像)发送至服务器，服务器可以基 于接收到的参数，得到处理结果等)，并将处理结果返回至至终端。
关于终端以及服务器的描述可以上述实施例的描述，这里不再赘述。
如图4示出了使用一项云平台提供的语言驱动的图像处理类云服务的流程。
1.开通并购买内容审核服务。
2.用户可以下载内容审核服务对应的软件开发工具包(software development kit，SDK)，通常云平台提供多个开发版本的SDK，供用户根据开发环境的需求选择，例如JAVA版本的SDK、python版本的SDK、PHP版本的SDK、Android版本的SDK等。
3.用户根据需求下载对应版本的SDK到本地后，将SDK工程导入至本地开发环境，在本地开发环境中进行配置和调试，本地开发环境还可以进行其他功能的开发，使得形成一个集合了语言驱动的图像处理类能力的应用。
4.语言驱动的图像处理类应用在被使用的过程中，当需要进行语言驱动的图像处理时，可以触发语言驱动的图像处理的API调用。当应用触发语言驱动的图像处理功能时，发起API请求至云环境中的语言驱动的图像处理类服务的运行实例，其中，API请求中携带图像，由云环境中的运行实例对图像进行处理，获得处理结果。
5.云环境将处理结果返回至应用，由此完成一次的本申请实施例提供的方法调用。
由于本申请实施例涉及大量神经网络的应用，为了便于理解，下面先对本申请实施例涉及的相关术语及神经网络等相关概念进行介绍。
(1)神经网络
神经网络可以是由神经单元组成的，神经单元可以是指以xs(即输入数据)和截距1为输入的运算单元，该运算单元的输出可以为：
其中，s＝1、2、……n，n为大于1的自然数，Ws为xs的权重，b为神经单元的偏置。f为神经单元的激活函数(activation functions)，用于将非线性特性引入神经网络中，来将神经单元中的输入信号转换为输出信号。该激活函数的输出信号可以作为下一层卷积层的输入，激活函数可以是sigmoid函数。神经网络是将多个上述单一的神经单元联结在一起形成的网络，即一个神经单元的输出可以是另一个神经单元的输入。每个神经单元的输入可以与前一层的局部接受域相连，来提取局部接受域的特征，局部接受域可以是由若干个神经单元组成的区域。
(2)transformer层
参照图5，图5为一种transformer层的架构示意，如图5所示，神经网络包括嵌入层和至少一个transformer层，至少一个transformer层可以为N个transformer层(N大于0的整数)，其中，每个transformer层包括依次相邻的注意力层、加和与归一化(add&norm)层、前馈(feed forward)层和加和与归一化层。在嵌入层，对当前输入进行嵌入处理，得到多个嵌入向量；在所述注意力层，从所述第一transformer层的上一层获取P个输入向量，以P个输入向量中的任意的第一输入向量为中心，基于预设的注意力窗口范围内的各个输入向量与该第一输入向量之间的关联度，得到该第一输入向量对应的中间向量，如此确定出P个输入向量对应的P个中间向量；在所述池化层，将所述P个中间向量合并为Q个输出向量，其中transformer层中最后一个transformer层得到的多个输出向量用作所述当前输入的特征表示。
(3)注意力机制(attention mechanism)
注意力机制模仿了生物观察行为的内部过程，即一种将内部经验和外部感觉对齐从而增加部分区域的观察精细度的机制，能够利用有限的注意力资源从大量信息中快速筛选出高价值信息。注意力机制可以快速提取稀疏数据的重要特征，因而被广泛用于自然语言处理任务，特别是机器翻译。而自注意力机制(self-attention mechanism)是注意力机制的改进，其减少了对外部信息的依赖，更擅长捕捉数据或特征的内部相关性。注意力机制的本质思想可以改写为如下公式：
其中，Lx＝||Source||代表Source的长度，公式含义即将Source中的构成元素想象成是由一系列的数据 对构成，此时给定目标Target中的某个元素Query，通过计算Query和各个Key的相似性或者相关性，得到每个Key对应Value的权重系数，然后对Value进行加权求和，即得到了最终的Attention数值。所以本质上Attention机制是对Source中元素的Value值进行加权求和，而Query和Key用来计算对应Value的权重系数。从概念上理解，把Attention可以理解为从大量信息中有选择地筛选出少量重要信息并聚焦到这些重要信息上，忽略大多不重要的信息。聚焦的过程体现在权重系数的计算上，权重越大越聚焦于其对应的Value值上，即权重代表了信息的重要性，而Value是其对应的信息。自注意力机制可以理解为内部Attention(intra attention)，Attention机制发生在Target的元素Query和Source中的所有元素之间，自注意力机制指的是在Source内部元素之间或者Target内部元素之间发生的Attention机制，也可以理解为Target＝Source这种特殊情况下的注意力计算机制，其具体计算过程是一样的，只是计算对象发生了变化而已。
(4)自然语言处理(natural language processing，NLP)
自然语言(natural language)即人类语言，自然语言处理(NLP)就是对人类语言的处理。自然语言处理是以一种智能与高效的方式，对文本数据进行系统化分析、理解与信息提取的过程。通过使用NLP及其组件，可以管理非常大块的文本数据，或者执行大量的自动化任务，并且解决各式各样的问题，如自动摘要(automatic summarization)，机器翻译(machine translation，MT)，命名实体识别(named entity recognition，NER)，关系提取(relation extraction，RE)，信息抽取(information extraction，IE)，情感分析，语音识别(speech recognition)，问答系统(question answering)以及主题分割等等。
(5)预训练语言模型(pre-trained language model)
预训练语言模型是一个自然语言序列编码器，为自然语言序列中的每个词进行编码成为一个向量表示，从而进行预测任务。它的训练包含两个阶段。在预训练(pre-training)阶段，该模型在大规模无监督文本上进行语言模型任务的训练，从而学习到一个词表示。在微调(finetuning)阶段，该模型利用预训练阶段学到的参数做初始化，在文本分类(text classification)，序列标注(sequence labeling)等下游任务(downstream task)上进行较少步骤的训练，就可以成功把预训练得到的语义信息成功迁移到下游任务上来。
(6)反向传播算法
卷积神经网络可以采用误差反向传播(back propagation，BP)算法在训练过程中修正初始的超分辨率模型中参数的大小，使得超分辨率模型的重建误差损失越来越小。具体地，前向传递输入信号直至输出会产生误差损失，通过反向传播误差损失信息来更新初始的超分辨率模型中参数，从而使误差损失收敛。反向传播算法是以误差损失为主导的反向传播运动，旨在得到最优的超分辨率模型的参数，例如权重矩阵。
(7)损失函数
在训练深度神经网络的过程中，因为希望深度神经网络的输出尽可能的接近真正想要预测的值，所以可以通过比较当前网络的预测值和真正想要的目标值，再根据两者之间的差异情况来更新每一层神经网络的权重向量(当然，在第一次更新之前通常会有初始化的过程，即为深度神经网络中的各层预先配置参数)，比如，如果网络的预测值高了，就调整权重向量让它预测低一些，不断地调整，直到深度神经网络能够预测出真正想要的目标值或与真正想要的目标值非常接近的值。因此，就需要预先定义“如何比较预测值和目标值之间的差异”，这便是损失函数(loss function)或目标函数(objective function)，它们是用于衡量预测值和目标值的差异的重要方程。其中，以损失函数举例，损失函数的输出值(loss)越高表示差异越大，那么深度神经网络的训练就变成了尽可能缩小这个loss的过程。
语言驱动的精确实例分割，是一种特殊的语义分割技术，指根据自然语言指导，在一幅图片中精确地分割出语言所描述的实例目标，其特点在于：1)传统语义分割模型为为属于同个类别的所有目标都预测同样的标签，并不区分同个类别中不同的目标，而语言驱动的精确实例分割，需要从多个同类目标中精确地识别出语言描述所对应的一个实例目标；2)语义分割模型需要预先定义一组语义类别的标签，从而学习分割出这些类别的目标，而语言驱动的精确实例分割可以接收更加灵活的自然语言输入，且不限目标类别。
由于自然语言输入的灵活性，语言驱动的实例分割方法主要依靠将自然语言句子编码和图像的视觉 编码进行融合，从而在视觉特征图上激活和语言编码相关性高的区域，然而这种跨模态特征融合的方案其挑战主要在于两个方面，首先是实例目标定位不准确，在多个拥挤的同类目标中，无法准确地锁定单个实例目标；其次是预测的掩膜不够准确，容易溢出到相邻的同类目标上。
为了解决上述问题，本申请实施例提供了一种数据处理方法。下面结合附图对本申请实施例的数据处理方法进行详细的介绍。
参照图6，图6为本申请实施例提供的一种数据处理方法的流程示意，如图6所示，本申请实施例提供的一种数据处理方法，可以包括步骤601至603，下面分别对这些步骤进行详细的描述。
601、获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域。
在一种可能的实现中，文本的语义可以指示从图像中确定目标对象对应的掩码区域(图像分割任务)或者是检测框(目标检测任务)。
在一种可能的实现中，所述文本的语义用于描述所述目标对象的特征。例如，图像中包括两个花瓶，分别是红色和黄色，文本可以为红色的花瓶，例如，图像中包括两个花瓶，分别处于图像的左侧和右侧，文本可以为左侧的花瓶。
在一种可能的实现中，在获取到图像以及文本之后，可以对图像和文本进行特征提取以及对齐，得到图像对应的第一图像特征以及文本对应的文本特征。
在一种可能的实现中，所述获取图像对应的第一图像特征以及文本对应的文本特征，具体包括：通过图像编码器处理所述图像，得到所述图像对应的图像特征，通过文本编码器处理所述文本，得到所述文本对应的第一文本特征，并通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
示例性的，图像编码器fv(或者可以称之为视觉编码器)可以对于给定的视觉图像I，采用Swin Transformer作为视觉编码器去提取多层级的视觉特征，将Swin Transformer中多个阶段(以四个阶段为例)产生的多尺度视觉特征，并记为
示例性的，文本编码器fl(或者可以称之为语言编码器)可以使用多层BERT(以12层为例)作为语言编码器，对于给定的语言表达W＝{wi}i＝1,…,T，其中T是表达式的长度。首先使用基于WordPiece的BERT分词器获得嵌入词向量E，将E送入BERT的隐藏编码层中去提取语言特征，为了能和Swin Transformer中四个阶段输出的视觉特征进行对齐，可以将BERT中每三层聚合，定义为一个阶段，这样就可以获得和视觉特征数量相同的语言特征，记为参照图7，同时可以在后面也会利用[CLS]标记对应的语言特征，使其作为一个全局的语义表达向量，记为Lg。
在一种可能的实现中，可以使用文本-像素对齐Word-pixel alignment模块来实现双向注意力机制，Word-pixel alignment模块是一种跨模态带门限的双向注意力模块，在图像和句子的编码阶段，促使视觉和语言特征在特征空间去对齐。可学习的特征门限机制，是为了防止更新融合后的特征时淹没原始特征信息。Word-pixel alignment的对齐效果可以如图8所示，通过Word-Pixel Alignment模块，在视觉和语言信息编码阶段，将语言信息融入到视觉编码中，同时将视觉信息融入到语言编码中，从而使得句子的单词特征和图片中对应的像素特征在跨模态特征空间建立相关性。
在一种可能的实现中，跨模态双向注意力模块BiAttn将视觉和语言特征在特征空间进行信息交互。使用该模块来融合双编码器每一阶段的视觉语言特征。可选的，其操作定义如下：
V′i,L′i＝BiAttn(Vi,Li),i∈{1,…,4}；
BiAttn函数的具体计算过程如下：


其中dk是视觉语言联合嵌入空间的维度，Wv,Wl,W′v,W′l均为投影矩阵。可选的，为了防止融合特征V′i,L′i会完全覆盖掉原有的特征Vi和Li的信息，设计了一个多层感知器MLP当做GATE来控制融合特征流入的信息量大小：
V′i←Gate(V′i),L′i←Gate(Li′)；
F′i＝GATE(Fi)＝MLP(Fi)⊙Fi；
其中Fi表示来在BCA模块的融合特征，F′i表示被抑制过后的融合特征，⊙表示矩阵逐元素乘法。MLP是一个两层感知机，第一层是线性层，随后是ReLU激活函数，第二层是线性层，随后是双曲正切激活函数。
在一种可能的实现中，为了更好的捕捉高层语义和生成像素级别的融合特征，可以使用一个多头注意层来融合来自视觉语言编码器的高层特征。首先，将高层的视觉特征Vo和语言特征Lo都投影到相同的特征空间，然后将它们拼接成一个融合特征Fo，其之后被送入交叉注意层。在拼接之前，将一个可学习的位置向量ep加在投影后的视觉特征上。最后交叉注意层输出特征So。上述运算可以被表述为如下公式：

So＝CrossAttn(Fo)+Vo,Fo＝[V′o；L′o],；
其中为投影矩阵，[；]为张量拼接操作。
在一种可能的实现中，为了将像素级别的特征上采样到原图大小获得最终的分割图，可以构建一个分割头。示例性的，分割头的输入可以为So和多尺度视觉特征然后获得如下输出：
其中ρ是一个两层的卷积网络，每一层都是一个3×3的卷积加上ReLU和批归一化，Up表示双线性插值上采样，γ表示1*1的卷积，将的每个像素进行特征投影，分割头的输出记为Y1。
602、根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征。
其中，神经网络可以根据第一图像特征，将多个第一嵌入向量处理成多个第二嵌入向量，其中，每个第二嵌入向量可以对应于目标对象的一个候选区域，不同的第二嵌入向量可以对应于目标对象的不同或存在重叠的候选区域。
相当于将图像特征从像素粒度变为以目标对象为粒度，也就是在跨模态特征融合中引入目标整体性约束，将属于同个目标的像素当做一个整体来和语言编码进行融合，以目标为单位来激活实例区域，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而提升分割精度。
在一种可能的实现中，所述神经网格包括多个transformer层。
示例性的，以图像分割为例，Sentence-object alignment模块，首先基于word-pixel align好的特征，生成可能的目标masks，再将自然语言句子特征和目标masks进行对齐align，从而更加精确地定位目标实例，如图9所示。本申请实施例设计了一个掩膜生成器MaskGenerator，基于编码器的输出So预测N个可能的目标掩膜。掩膜生成器由一个6层的Transformer解码器组成。其输入为So和N可学习的第一嵌入向量(也就是查询向量Q)，输出为N个第二嵌入向量(也就是目标掩膜特征向量编码Qo)，即：
Qo＝MaskGenerator(Q,So).；
603、根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域。
由于多个第二嵌入向量中仅有部分是对应于目标对象的，在得到多个第二嵌入向量之后，可以根据文本特征为每个第二嵌入向量分配一个权值，权值较高的第二嵌入向量最有可能包含文本所指的目标对象。
示例性的，以图像分割任务为例，可以根据文本特征Lg为Qo的每个掩码向量(第二嵌入向量)分配一个权值Qw，Qw中较高的权重表明相应的掩码最有可能包含语言所指的对象。随后通过将Qo和Y1相乘得到N个掩码预测YN。最后，将YN和Qw相乘得到最后的掩码预测M，这个过程可以表示如下：
Qw＝softmax(sim(Lg,Qo)),
其中，sim(,)表示余弦相似度函数，表示广播张量乘法。
图7为本申请实施例的一个网络架构示意，其中，其整体架构设计遵寻了经典的编码器—解码器的范式，编码器部分由一个视觉编码器和一个语言编码器来提取视觉和语言特征，Word-Pixel Alignment(WPA)模块在视觉和语言编码的中间层，实现跨模态交互。然后使用一个交叉注意层将视觉和语言编码器的输出做跨模态融合。解码器部分由一个生成N个掩码查询向量的掩码生成器，一个将像素特征上采样的分割头和一个Sentence-Object Alignment模块(SOA)，其根据语句特征对输出的掩码查询向量赋权，同时加该利用该权重，对分割头生成的分割特征进行加权求和获得最终的分割掩码。
针对于上述模型的训练过程，以图像分割任务为例，对于参考图像分割而言，图像中的每一个像素都要被归类到前景或者背景，因此该任务可以被视为一个像素级别的二分类任务。具体来说，使用双线性插值将M上采样到原图大小，得到M′。令M′和(真实掩码)每一点i的值分别为m′i和分割损失形式如下：
其中σ表示sigmoid函数，j表示训练批次中的第j个图像。
此外，为了增强模型分离前景和背景的能力，可以使用一个像素级的对比损失函数来作为分割损失的辅助函数。该函数减小目标物体内的像素特征的距离，增大目标物体内像素特征和物体外像素特征的距离，如图10所示。
将真实掩码放缩到和分割特征图Y1一样的大小。令Y1中每一点i的向量为yi。然后利用的先验将yi划分出正负样本。对于每一点i而言，如果等于0，那么yi属于负样本集合N，记为否则yi属于正样本集合P，记为之后将正样本和负样本的均值向量分别记为和辅助损失函数形式如下：


其中τ是温度系数，最后，总损失函数如下：
模型训练结束后，在推理阶段，输入一幅图片，以及一个自然语言句子描述图片中的一个实例目标，模型会直接预测出该实例目标的掩膜M或者检测框，将其上采样插值回原图大小，并进行二值化，即可分割出实例目标。
接下来结合试验介绍本申请的有益效果：
参考图像分割常用的数据集有三个分别为：RefCOCO，RefCOCO+和RefCOCOg(也被称为G-Ref)。这三个数据集的图像都是来自于MSCOCO数据集，并各自配上了不同的语言标注。RefCOCO和RefCOCO+的语言标注是通过一个名为ReferitGame的游戏生成的，RefCOCO由142,209个自然语言标注和19,994张图片构成，RefCOCO+由141,564个自然语言标注和19,992张图片构成。RefCOCO和RefCOCO+两个数据集的主要区别在于RefCOCO+不允许在语言标注中使用类似“left”，“front”之类的定位词。因此，相较于RefCOCO而言，RefCOCO+数据集更加具有挑战性。G-Ref数据集上的语言标注是来源于Amazon Mechanical Truk，它包含有85,474个语言标注和26,711张图片。同时该数据集有两种划分方式，分别是UMD划分和Google划分。相较于RefCOCO和RefCOCO+而言，G-Ref的语言标注更加复杂多变，其句子的平均长度也是要大于RefCOCO和RefCOCO+数据集上句子的平均长度，这就使得G-Ref成为一个更加有挑战性的数据集。
原始输入数据为RGB图像，0-1掩码矩阵，语言标注字符串。其中图像部分数据的预处理为：对于训练数据将RGB图像归一化，正则化之后，通过双线性插值放缩到统一的分辨率448*448。同时0-1掩码矩阵经过最邻近插值放缩到和RGB图像一致的分辨率。对于测试数据只需要对RGB图像进行上述处理，不需要对0-1掩码矩阵进行最邻近插值。语言部分数据的预处理为：使用HuggingFace库中的BertTokenizer，将输入字符串标记化。BertTokenizer是基于WordPiece嵌入方法，其字典的大小为30,000。对于每一个标记化后的序列，第一个标记都是一个特殊的[CLS]标记。如果输入的是多个句子，那么在句子与句子之间会插入另一个特殊的[SEP]标记。
采用参考图像分割常用的三种度量方法来评估模型的性能，分别是全局交并比(oIoU)，平均交并比(mIoU)和prec@X。这三种度量方法都是目标识别的常用度量指标交并比(IoU)，其主要用来表示预测区域和真实区域的相似程度。在参考图像分割中，交并比的求法可以简化如下：给定预测掩码M和真实掩码其交并比定义如下：
全局交并比是所有测试图像的交集之和比上并集之和，平均交并比是所有测试图像交并比的平均值，prec@X是交并比大于某一阈值X的图像占所有测试图像的百分比，实验中X的取值通常为0.5，0.6，0.7，0.8，0.9。
表1与现有技术方法在RefCOCO和G-Ref两个数据集上的比较

在表1中，将CoupAlign与之前SOTA方法在RefCOCO和G-Ref上比较oIoU。RefCOCO数据集提供的语言标注，包含了许多位置词语，例如：“The closest girl on the right”。这要求模型不仅需要理解名词与物体之间的对应关系，还需要理解方位词表示的物体间的位置关系。与最新的SOTA方法LAVT比较，CoupAlign在RefCOCO的val，testA和testB上分别提升了1.97％，1.94％，1.79％。G-Ref上的语言标注相较于RefCOCO有更为复杂的语法结构，同时平均句子长度也更长。例如：“chocolate dessert directly in front of us in the center”这类句子需要更加细粒度的单词—像素对齐，更加准确的句子—掩码对齐，更加全面地理解语言。如表1所示在G-Ref上，CoupAlign在val和test上超过LAVT大约1.6％和0.13％。两个数据集的各个子数据集上的结果均超过现有的SOTA，证明了方法的有效性。
单词像素对齐使得跨模态的交互同时发生在编码的底层和高层阶段。在表2中，可以发现，在移除单词像素对齐模块之后，模型在oIoU这一指标上面下降了大约4.3％，这一点表明编码阶段的单词像素对齐模块的存在十分必要。同时当将双向注意力机制替换为单向注意力机制时，模型在oIoU这一指标上面下降了2％左右。这表明不仅是从语言到视觉的注意，从视觉到语言的注意力也十分的重要。当句子目标对齐模块移除之后，模型的oIoU指标下降了大约1.7％。这一点表明句子目标对齐模块对掩码的约束的效果，有助于提高CoupAlign预测的准确性。在表2的最后两行展示了移除辅助损失函数之后的对比效果，在移除辅助损失函数之后，模型在oIOU指标上下降了大约1％，这证明了辅助损失函数增强模型前景和背景分离能力有助于CoupAlign更好得进行多层次的对齐。
表2消融实验
为了验证的单词像素对齐模块和句子目标对齐模块可以提供准确并且持久的对齐。在图11A中可视化了对齐模块中间层的注意力图，如图11A所示，可以看见单词像素对齐模块高亮了与单词语义最相关的像素区域。值得注意的是，参考图像分割的语言标注的词汇库比传统的语义分割的词汇库要大很多。在这种情况下，CoupAlign不仅能够捕捉区分不同词性的词汇，同时对同义名词也有比较好的分辨能力。例如：其可以分辨“child”，“man”，“lady”等同义名词，但是这些在语义分割的数据集中常常都模糊地被定义为“person”类别。关于句子目标对齐模块，按照与句子语义相似度从大到小的顺序可视化了掩码预测。相似度越大的掩码与目标物体重合度越大，相似度越小的掩码与目标物体重合越小。在图11A中，可以看到句子目标对齐模块，使得模型可以关注到不同物体，从而感知到物体间的位置关系。并且因为引入了目标整体性约束，使得模型的分割预测较少得产生镂空，破碎等现象。
在图11B中，可视化模型最终预测结果的一些例子。CoupAlign在同类目标拥挤的场景下具有很强的定位能力。
本申请实施例提供了一种数据处理方法，所述方法包括：获取图像对应的第一图像特征以及文本对 应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域。相当于将图像特征从像素粒度变为以目标对象为粒度，也就是在跨模态特征融合中引入目标整体性约束，将属于同个目标的像素当做一个整体来和语言编码进行融合，以目标为单位来激活实例区域，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而提升模型的处理精度。
此外，本申请还提供了一种数据处理方法，所述方法包括：
获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；所述第一图像特征以及所述文本特征为根据特征提取网络得到的；
根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域；
根据所述预测区域和所述图像中所述目标对象对应的真实区域之间的差异，更新所述特征提取网络以及所述神经网络。
在一种可能的实现中，所述预测区域为掩码区域或者检测框。
在一种可能的实现中，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
在一种可能的实现中，所述获取图像对应的第一图像特征以及文本对应的文本特征，包括：
通过图像编码器处理所述图像，得到所述图像对应的图像特征；
通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
参照图12，图12为本申请实施例提供的一种数据处理装置的结构示意，如图12所示，本申请实施例提供的一种数据处理装置1200，包括：
处理模块1201，用于获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；
根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域。
其中，关于处理模块1201的具体描述可以参照上述实施例中步骤601至603的描述，这里不再赘述。
其中，图像可以包括目标对象在内的多个对象，每个第二嵌入向量对应于所述图像中的一个对象，多个第二嵌入向量中的一个或多个嵌入向量可以对应于目标对象。应理解，这里的“对应”可以理解为，第二嵌入向量用于描述所述图像中的一个对象的特征，通过神经网络得到的第二嵌入向量可以将图像中的不同对象进行区分，以便后续的预测过程中能够以对象为粒度。
相当于将图像特征从像素粒度变为以目标对象为粒度，也就是在跨模态特征融合中引入目标整体性约束，将属于同个目标的像素当做一个整体来和语言编码进行融合，以目标为单位来激活实例区域，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而 提升模型的处理精度。
在一种可能的实现中，所述预测区域为掩码区域或者检测框。
在一种可能的实现中，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
在一种可能的实现中，所述处理模块，具体用于：
通过图像编码器处理所述图像，得到所述图像对应的图像特征；
通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
在一种可能的实现中，所述第一图像特征为上采样到和所述图像的尺寸一致的特征。
在一种可能的实现中，所述神经网格包括多个transformer层。
此外本申请实施例还提供了一种数据处理装置，包括：
处理模块，用于获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；所述第一图像特征以及所述文本特征为根据特征提取网络得到的；
根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域；
更新模块，用于根据所述预测区域和所述图像中所述目标对象对应的真实区域之间的差异，更新所述特征提取网络以及所述神经网络。
其中，图像可以包括目标对象在内的多个对象，每个第二嵌入向量对应于所述图像中的一个对象，多个第二嵌入向量中的一个或多个嵌入向量可以对应于目标对象。应理解，这里的“对应”可以理解为，第二嵌入向量用于描述所述图像中的一个对象的特征，通过神经网络得到的第二嵌入向量可以将图像中的不同对象进行区分，以便后续的预测过程中能够以对象为粒度。
相当于将图像特征从像素粒度变为以目标对象为粒度，也就是在跨模态特征融合中引入目标整体性约束，将属于同个目标的像素当做一个整体来和语言编码进行融合，以目标为单位来激活实例区域，可以有效解决现有语言驱动的精确实例分割方法目标定位和掩膜预测或者检测框预测不准确的问题，从而提升模型的处理精度。
在一种可能的实现中，所述预测区域为掩码区域或者检测框。
在一种可能的实现中，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
在一种可能的实现中，所述处理模块，具体用于：
通过图像编码器处理所述图像，得到所述图像对应的图像特征；
通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
接下来介绍本申请实施例提供的一种执行设备，请参阅图13，图13为本申请实施例提供的执行设备的一种结构示意图，执行设备1300具体可以表现为虚拟现实VR设备、手机、平板、笔记本电脑、智能穿戴设备、监控数据处理设备或服务器等，此处不做限定。具体的，执行设备1300包括：接收器1301、发射器1302、处理器1303和存储器1304(其中执行设备1300中的处理器1303的数量可以一个或多个，图13中以一个处理器为例)，其中，处理器1303可以包括应用处理器13031和通信处理器13032。在本申请的一些实施例中，接收器1301、发射器1302、处理器1303和存储器1304可通过总线或其它方式连接。
存储器1304可以包括只读存储器和随机存取存储器，并向处理器1303提供指令和数据。存储器1304 的一部分还可以包括非易失性随机存取存储器(non-volatile random access memory，NVRAM)。存储器1304存储有处理器和操作指令、可执行模块或者数据结构，或者它们的子集，或者它们的扩展集，其中，操作指令可包括各种操作指令，用于实现各种操作。
处理器1303控制执行设备的操作。具体的应用中，执行设备的各个组件通过总线系统耦合在一起，其中总线系统除包括数据总线之外，还可以包括电源总线、控制总线和状态信号总线等。但是为了清楚说明起见，在图中将各种总线都称为总线系统。
上述本申请实施例揭示的方法可以应用于处理器1303中，或者由处理器1303实现。处理器1303可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器1303中的硬件的集成逻辑电路或者软件形式的指令完成。上述的处理器1303可以是通用处理器、数字信号处理器(digital signal processing，DSP)、微处理器或微控制器，还可进一步包括专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field-programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。该处理器1303可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存储器，闪存、只读存储器，可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器1304，处理器1303读取存储器1304中的信息，结合其硬件完成上述方法中涉及模型推理过程的步骤。
接收器1301可用于接收输入的数字或字符信息，以及产生与执行设备的相关设置以及功能控制有关的信号输入。发射器1302可用于通过第一接口输出数字或字符信息；发射器1302还可用于通过第一接口向磁盘组发送指令，以修改磁盘组中的数据；发射器1302还可以包括显示屏等显示设备。
本申请实施例还提供了一种训练设备，请参阅图14，图14是本申请实施例提供的训练设备一种结构示意图，具体的，训练设备1400由一个或多个服务器实现，训练设备1400可因配置或性能不同而产生比较大的差异，可以包括一个或一个以上中央处理器(central processing units，CPU)1414(例如，一个或一个以上处理器)和存储器1432，一个或一个以上存储应用程序1442或数据1444的存储介质1430(例如一个或一个以上海量存储设备)。其中，存储器1432和存储介质1430可以是短暂存储或持久存储。存储在存储介质1430的程序可以包括一个或一个以上模块(图示没标出)，每个模块可以包括对训练设备中的一系列指令操作。更进一步地，中央处理器1414可以设置为与存储介质1430通信，在训练设备1400上执行存储介质1430中的一系列指令操作。
训练设备1400还可以包括一个或一个以上电源1426，一个或一个以上有线或无线网络接口1450，一个或一个以上输入输出接口1458；或，一个或一个以上操作系统1441，例如Windows ServerTM，Mac OS XTM，UnixTM,LinuxTM，FreeBSDTM等等。
本申请实施例中，中央处理器1414，用于执行上述实施例中和模型训练相关的动作。
本申请实施例中还提供一种包括计算机程序产品，当其在计算机上运行时，使得计算机执行如前述执行设备所执行的步骤，或者，使得计算机执行如前述训练设备所执行的步骤。
本申请实施例中还提供一种计算机可读存储介质，该计算机可读存储介质中存储有用于进行信号处理的程序，当其在计算机上运行时，使得计算机执行如前述执行设备所执行的步骤，或者，使得计算机执行如前述训练设备所执行的步骤。
本申请实施例提供的执行设备、训练设备或终端设备具体可以为芯片，芯片包括：处理单元和通信单元，所述处理单元例如可以是处理器，所述通信单元例如可以是输入/输出接口、管脚或电路等。该处理单元可执行存储单元存储的计算机执行指令，以使执行设备内的芯片执行上述实施例描述的数据处理方法，或者，以使训练设备内的芯片执行上述实施例描述的数据处理方法。可选地，所述存储单元为所述芯片内的存储单元，如寄存器、缓存等，所述存储单元还可以是所述无线接入设备端内的位于所述芯片外部的存储单元，如只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)等。
具体的，请参阅图15，图15为本申请实施例提供的芯片的一种结构示意图，所述芯片可以表现为神经网络处理器NPU 1500，NPU 1500作为协处理器挂载到主CPU(Host CPU)上，由Host CPU分配任务。NPU的核心部分为运算电路1503，通过控制器1504控制运算电路1503提取存储器中的矩阵数据并进行乘法运算。
在一些实现中，运算电路1503内部包括多个处理单元(Process Engine,PE)。在一些实现中，运算电路1503是二维脉动阵列。运算电路1503还可以是一维脉动阵列或者能够执行例如乘法和加法这样的数学运算的其它电子线路。在一些实现中，运算电路1503是通用的矩阵处理器。
举例来说，假设有输入矩阵A，权重矩阵B，输出矩阵C。运算电路从权重存储器1502中取矩阵B相应的数据，并缓存在运算电路中每一个PE上。运算电路从输入存储器1501中取矩阵A数据与矩阵B进行矩阵运算，得到的矩阵的部分结果或最终结果，保存在累加器(accumulator)1508中。
统一存储器1506用于存放输入数据以及输出数据。权重数据直接通过存储单元访问控制器(Direct Memory Access Controller，DMAC)1505，DMAC被搬运到权重存储器1502中。输入数据也通过DMAC被搬运到统一存储器1506中。
BIU为Bus Interface Unit即，总线接口单元1510，用于AXI总线与DMAC和取指存储器(Instruction Fetch Buffer，IFB)1509的交互。
总线接口单元1510(Bus Interface Unit，简称BIU)，用于取指存储器1509从外部存储器获取指令，还用于存储单元访问控制器1505从外部存储器获取输入矩阵A或者权重矩阵B的原数据。
DMAC主要用于将外部存储器DDR中的输入数据搬运到统一存储器1506或将权重数据搬运到权重存储器1502中或将输入数据数据搬运到输入存储器1501中。
向量计算单元1507包括多个运算处理单元，在需要的情况下，对运算电路1503的输出做进一步处理，如向量乘，向量加，指数运算，对数运算，大小比较等等。主要用于神经网络中非卷积/全连接层网络计算，如Batch Normalization(批归一化)，像素级求和，对特征平面进行上采样等。
在一些实现中，向量计算单元1507能将经处理的输出的向量存储到统一存储器1506。例如，向量计算单元1507可以将线性函数；或，非线性函数应用到运算电路1503的输出，例如对卷积层提取的特征平面进行线性插值，再例如累加值的向量，用以生成激活值。在一些实现中，向量计算单元1507生成归一化的值、像素级求和的值，或二者均有。在一些实现中，处理过的输出的向量能够用作到运算电路1503的激活输入，例如用于在神经网络中的后续层中的使用。
控制器1504连接的取指存储器(instruction fetch buffer)1509，用于存储控制器1504使用的指令；
统一存储器1506，输入存储器1501，权重存储器1502以及取指存储器1509均为On-Chip存储器。外部存储器私有于该NPU硬件架构。
其中，上述任一处提到的处理器，可以是一个通用中央处理器，微处理器，ASIC，或一个或多个用于控制上述程序执行的集成电路。
另外需说明的是，以上所描述的装置实施例仅仅是示意性的，其中所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部模块来实现本实施例方案的目的。另外，本申请提供的装置实施例附图中，模块之间的连接关系表示它们之间具有通信连接，具体可以实现为一条或多条通信总线或信号线。
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到本申请可借助软件加必需的通用硬件的方式来实现，当然也可以通过专用硬件包括专用集成电路、专用CPU、专用存储器、专用元器件等来实现。一般情况下，凡由计算机程序完成的功能都可以很容易地用相应的硬件来实现，而且，用来实现同一功能的具体硬件结构也可以是多种多样的，例如模拟电路、数字电路或专用电路等。但是，对本申请而言更多情况下软件程序实现是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在可读取的存储介质中，如计算机的软盘、U盘、移动硬盘、ROM、RAM、磁碟或者光盘等，包括若干指令用以使得一台计算机设备(可以是个人计算机，训练设备，或者网络设备等)执行本申请各个实施例所述的方 法。
在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。
所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、训练设备或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、训练设备或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存储的任何可用介质或者是包含一个或多个可用介质集成的训练设备、数据中心等数据存储设备。所述可用介质可以是磁性介质，(例如，软盘、硬盘、磁带)、光介质(例如，DVD)、或者半导体介质(例如固态硬盘(Solid State Disk，SSD))等。

Claims (23)

  1. 一种数据处理方法，其特征在于，包括：
    获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；
    根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
    根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域。
  2. 根据权利要求1所述的方法，其特征在于，所述预测区域为掩码区域或者检测框。
  3. 根据权利要求1或2所述的方法，其特征在于，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
  4. 根据权利要求1至3任一所述的方法，其特征在于，所述获取图像对应的第一图像特征以及文本对应的文本特征，包括：
    通过图像编码器处理所述图像，得到所述图像对应的图像特征；
    通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
    通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
  5. 根据权利要求1至4任一所述的方法，其特征在于，所述第一图像特征为上采样到和所述图像的尺寸一致的特征。
  6. 根据权利要求1至5任一所述的方法，其特征在于，所述神经网格包括多个transformer层。
  7. 一种数据处理方法，其特征在于，包括：
    获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；所述第一图像特征以及所述文本特征为根据特征提取网络得到的；
    根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
    根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域；
    根据所述预测区域和所述图像中所述目标对象对应的真实区域之间的差异，更新所述特征提取网络以及所述神经网络。
  8. 根据权利要求7所述的方法，其特征在于，所述预测区域为掩码区域或者检测框。
  9. 根据权利要求7或8所述的方法，其特征在于，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
  10. 根据权利要求7至9任一所述的方法，其特征在于，所述获取图像对应的第一图像特征以及文 本对应的文本特征，包括：
    通过图像编码器处理所述图像，得到所述图像对应的图像特征；
    通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
    通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
  11. 一种数据处理装置，其特征在于，包括：
    处理模块，用于获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；
    根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
    根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域。
  12. 根据权利要求11所述的装置，其特征在于，所述预测区域为掩码区域或者检测框。
  13. 根据权利要求11或12所述的装置，其特征在于，所述文本的语义对应于目标对象，具体包括：所述文本的语义用于描述所述目标对象的特征。
  14. 根据权利要求11至13任一所述的装置，其特征在于，所述处理模块，具体用于：
    通过图像编码器处理所述图像，得到所述图像对应的图像特征；
    通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
    通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
  15. 根据权利要求11至14任一所述的装置，其特征在于，所述第一图像特征为上采样到和所述图像的尺寸一致的特征。
  16. 根据权利要求11至15任一所述的装置，其特征在于，所述神经网格包括多个transformer层。
  17. 一种数据处理装置，其特征在于，包括：
    处理模块，用于获取图像对应的第一图像特征以及文本对应的文本特征；所述文本的语义对应于目标对象，且所述文本指示从所述图像中预测目标对象对应的区域；所述第一图像特征以及所述文本特征为根据特征提取网络得到的；
    根据预设的多个第一嵌入向量以及所述第一图像特征，通过神经网络，得到多个第二嵌入向量，每个第二嵌入向量对应于所述图像中的一个对象；每个所述第二嵌入向量和所述第一图像特征用于融合得到一个对应的第二图像特征；
    根据所述文本特征和所述多个第二嵌入向量之间的相似度，确定每个所述第二嵌入向量对应的权重，多个所述权重用于和所述多个第二图像特征进行融合，以确定所述目标对象对应的预测区域；
    更新模块，用于根据所述预测区域和所述图像中所述目标对象对应的真实区域之间的差异，更新所述特征提取网络以及所述神经网络。
  18. 根据权利要求17所述的装置，其特征在于，所述预测区域为掩码区域或者检测框。
  19. 根据权利要求17或18所述的装置，其特征在于，所述文本的语义对应于目标对象，具体包括： 所述文本的语义用于描述所述目标对象的特征。
  20. 根据权利要求17至19任一所述的装置，其特征在于，所述处理模块，具体用于：
    通过图像编码器处理所述图像，得到所述图像对应的图像特征；
    通过文本编码器处理所述文本，得到所述文本对应的第一文本特征；
    通过双向注意力机制融合所述第三图像特征以及所述第一文本特征，得到所述图像对应的第一图像特征以及所述文本对应的文本特征。
  21. 一种计算机存储介质，其特征在于，所述计算机存储介质存储有一个或多个指令，所述指令在由一个或多个计算机执行时使得所述一个或多个计算机执行权利要求1至10中任一项所述方法的操作。
  22. 一种计算机程序产品，其特征在于，包括计算机可读指令，当所述计算机可读指令在计算机设备上运行时，使得所述计算机设备执行如权利要求1至10任一所述的方法。
  23. 一种系统，包括至少一个处理器，至少一个存储器；所述处理器、所述存储器通过通信总线连接并完成相互间的通信；
    所述至少一个存储器用于存储代码；
    所述至少一个处理器用于执行所述代码，以执行如权利要求1至10任一所述的方法。
PCT/CN2023/124977 2022-10-20 2023-10-17 一种数据处理方法及其装置 Ceased WO2024083121A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP23879109.9A EP4592866B1 (en) 2022-10-20 2023-10-17 Data processing method and apparatus
US19/182,947 US20250272945A1 (en) 2022-10-20 2025-04-18 Data processing method and apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211292146.9A CN115757692A (zh) 2022-10-20 2022-10-20 一种数据处理方法及其装置
CN202211292146.9 2022-10-20

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/182,947 Continuation US20250272945A1 (en) 2022-10-20 2025-04-18 Data processing method and apparatus

Publications (1)

Publication Number Publication Date
WO2024083121A1 true WO2024083121A1 (zh) 2024-04-25

Family

ID=85352471

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/124977 Ceased WO2024083121A1 (zh) 2022-10-20 2023-10-17 一种数据处理方法及其装置

Country Status (4)

Country Link
US (1) US20250272945A1 (zh)
EP (1) EP4592866B1 (zh)
CN (1) CN115757692A (zh)
WO (1) WO2024083121A1 (zh)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118115622A (zh) * 2024-04-28 2024-05-31 腾讯科技（深圳）有限公司 图像生成模型的处理方法、装置、设备、存储介质及产品
CN118505858A (zh) * 2024-07-19 2024-08-16 山东海量信息技术研究院 一种图像生成方法、设备、介质及计算机程序产品
CN119228822A (zh) * 2024-09-05 2024-12-31 北京纳通医用机器人科技有限公司 图像分割方法、装置、设备及介质
CN119313891A (zh) * 2024-12-18 2025-01-14 浙江大华技术股份有限公司 目标识别方法、电子设备和计算机可读存储介质
CN119322982A (zh) * 2024-09-13 2025-01-17 同济大学 基于模态同步的多模态情感检测方法
CN119478423A (zh) * 2025-01-13 2025-02-18 北京科技大学 一种基于开放域的跨模态遥感图像目标分割方法及装置
CN120198392A (zh) * 2025-03-11 2025-06-24 虹纬盐城纺织有限公司 基于图像识别的超柔针织纱布料缺陷检测方法及系统
CN120495074A (zh) * 2025-07-21 2025-08-15 杭州筋斗云文化服务有限公司 一种基于深度神经网络的人像摄影图像仿色方法和系统

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115757692A (zh) * 2022-10-20 2023-03-07 华为技术有限公司 一种数据处理方法及其装置
CN116503594A (zh) * 2023-03-24 2023-07-28 华为技术有限公司 一种数据处理方法及其装置
CN116433621B (zh) * 2023-03-25 2026-02-13 华为技术有限公司 一种数据处理方法及其装置
US20240357104A1 (en) * 2023-04-21 2024-10-24 Nokia Technologies Oy Determining regions of interest using learned image codec for machines
CN116883715A (zh) * 2023-05-30 2023-10-13 华为技术有限公司 一种数据处理方法及其装置
CN116704506A (zh) * 2023-06-21 2023-09-05 大连理工大学 一种基于交叉环境注意力的指代图像分割方法
CN116993976B (zh) * 2023-07-17 2024-06-14 中国科学院自动化研究所 引用图像分割模型训练方法及引用图像分割方法
AU2023446714A1 (en) * 2023-08-04 2025-02-20 Shenzhen Mango Science And Technology Innovation Co., Ltd. Method for adjusting lighting device, apparatus, lighting device and storage medium
CN117251592A (zh) * 2023-08-23 2023-12-19 华为技术有限公司 一种数据处理方法及其装置
US20260030861A1 (en) * 2024-07-29 2026-01-29 Nvidia Corporation Segmentation of media content using vision language models
CN119314337A (zh) * 2024-12-17 2025-01-14 松立控股集团股份有限公司 一种动静态交通协同治理方法及系统
CN120747198B (zh) * 2025-06-23 2026-01-02 电子科技大学 基于计算机视觉的米粒测量方法
CN120599274B (zh) * 2025-08-08 2025-11-07 广东工业大学 一种基于深度学习的直肠癌医学影像病灶分割方法及装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108776787A (zh) * 2018-06-04 2018-11-09 北京京东金融科技控股有限公司 图像处理方法及装置、电子设备、存储介质
CN113065576A (zh) * 2021-02-26 2021-07-02 华为技术有限公司 一种特征提取的方法以及装置
CN113505193A (zh) * 2021-06-01 2021-10-15 华为技术有限公司 一种数据处理方法及相关设备
US20220147715A1 (en) * 2019-05-16 2022-05-12 Huawei Technologies Co., Ltd. Text processing method, model training method, and apparatus
CN115115913A (zh) * 2022-06-02 2022-09-27 北京科技大学 一种数据处理方法、装置、电子设备及存储介质
CN115757692A (zh) * 2022-10-20 2023-03-07 华为技术有限公司 一种数据处理方法及其装置

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110033022A (zh) * 2019-03-08 2019-07-19 腾讯科技（深圳）有限公司 文本的处理方法、装置和存储介质
CN111210443B (zh) * 2020-01-03 2022-09-13 吉林大学 基于嵌入平衡的可变形卷积混合任务级联语义分割方法
CN112215235B (zh) * 2020-10-16 2024-04-26 深圳华付技术股份有限公司 一种针对具有大字符间距与局部遮挡的场景文本检测方法
CN114283430B (zh) * 2021-12-03 2024-12-06 苏州大创科技有限公司 跨模态图文匹配训练方法及装置、存储介质、电子设备
CN114298057B (zh) * 2022-01-04 2024-08-09 中国人民解放军国防科技大学 一种基于数据增强的视觉语义嵌入方法及系统
CN114648631B (zh) * 2022-03-22 2024-11-01 平安科技（深圳）有限公司 图像描述生成方法和装置、电子设备及存储介质
CN115131556A (zh) * 2022-05-27 2022-09-30 吉林大学 一种基于深度学习的图像实例分割方法
CN115063585B (zh) * 2022-05-30 2025-08-29 华为技术有限公司 一种无监督语义分割模型的训练方法及相关装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108776787A (zh) * 2018-06-04 2018-11-09 北京京东金融科技控股有限公司 图像处理方法及装置、电子设备、存储介质
US20220147715A1 (en) * 2019-05-16 2022-05-12 Huawei Technologies Co., Ltd. Text processing method, model training method, and apparatus
CN113065576A (zh) * 2021-02-26 2021-07-02 华为技术有限公司 一种特征提取的方法以及装置
CN113505193A (zh) * 2021-06-01 2021-10-15 华为技术有限公司 一种数据处理方法及相关设备
CN115115913A (zh) * 2022-06-02 2022-09-27 北京科技大学 一种数据处理方法、装置、电子设备及存储介质
CN115757692A (zh) * 2022-10-20 2023-03-07 华为技术有限公司 一种数据处理方法及其装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4592866A4

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118115622A (zh) * 2024-04-28 2024-05-31 腾讯科技（深圳）有限公司 图像生成模型的处理方法、装置、设备、存储介质及产品
CN118505858A (zh) * 2024-07-19 2024-08-16 山东海量信息技术研究院 一种图像生成方法、设备、介质及计算机程序产品
CN119228822A (zh) * 2024-09-05 2024-12-31 北京纳通医用机器人科技有限公司 图像分割方法、装置、设备及介质
CN119322982A (zh) * 2024-09-13 2025-01-17 同济大学 基于模态同步的多模态情感检测方法
CN119322982B (zh) * 2024-09-13 2025-11-18 同济大学 基于模态同步的多模态情感检测方法
CN119313891A (zh) * 2024-12-18 2025-01-14 浙江大华技术股份有限公司 目标识别方法、电子设备和计算机可读存储介质
CN119478423A (zh) * 2025-01-13 2025-02-18 北京科技大学 一种基于开放域的跨模态遥感图像目标分割方法及装置
CN120198392A (zh) * 2025-03-11 2025-06-24 虹纬盐城纺织有限公司 基于图像识别的超柔针织纱布料缺陷检测方法及系统
CN120495074A (zh) * 2025-07-21 2025-08-15 杭州筋斗云文化服务有限公司 一种基于深度神经网络的人像摄影图像仿色方法和系统

Also Published As

Publication number Publication date
CN115757692A (zh) 2023-03-07
EP4592866A1 (en) 2025-07-30
EP4592866A4 (en) 2025-08-27
EP4592866B1 (en) 2026-03-11
US20250272945A1 (en) 2025-08-28

Similar Documents

Publication Publication Date Title
WO2024083121A1 (zh) 一种数据处理方法及其装置
WO2024041479A1 (zh) 一种数据处理方法及其装置
US20260080190A1 (en) Data processing method and related device
CN116306672A (zh) 一种数据处理方法及其装置
WO2024245063A1 (zh) 一种数据处理方法及其装置
WO2024260402A1 (zh) 一种数据处理方法及其装置
WO2025040048A1 (zh) 一种数据处理方法及其装置
WO2025026210A1 (zh) 一种数据处理方法及其装置
WO2024239983A1 (zh) 一种可控生成的方法及其装置
WO2024213099A1 (zh) 一种数据处理方法及其装置
US20260044732A1 (en) Data processing method and apparatus
WO2024245061A1 (zh) 一种数据处理方法及其装置
WO2024017287A1 (zh) 一种模型训练方法及其装置
WO2025016380A1 (zh) 一种数据处理方法及相关装置
CN112861474B (zh) 一种信息标注方法、装置、设备及计算机可读存储介质
WO2025130968A1 (zh) 一种数据处理方法及其装置
WO2025189851A9 (zh) 一种数据处理方法及其装置
WO2025044967A1 (zh) 一种数据处理方法及其装置
WO2025103049A1 (zh) 一种数据处理方法及其装置
WO2025067287A1 (zh) 一种视频编码方法及其装置
EP4546292A1 (en) Data processing method and device
WO2025031373A1 (zh) 一种数据处理方法及其装置
CN116052714A (zh) 一种数据处理方法及其装置
WO2025167446A1 (zh) 一种数据处理方法及其装置
WO2025124301A1 (zh) 一种文本处理方法及其装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23879109

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2023879109

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2023879109

Country of ref document: EP

Effective date: 20250425

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 2023879109

Country of ref document: EP

WWG Wipo information: grant in national office

Ref document number: 2023879109

Country of ref document: EP