WO2022007527A1 - 样本数据标注系统、方法以及相关设备 - Google Patents

样本数据标注系统、方法以及相关设备 Download PDF

Info

Publication number
WO2022007527A1
WO2022007527A1 PCT/CN2021/095786 CN2021095786W WO2022007527A1 WO 2022007527 A1 WO2022007527 A1 WO 2022007527A1 CN 2021095786 W CN2021095786 W CN 2021095786W WO 2022007527 A1 WO2022007527 A1 WO 2022007527A1
Authority
WO
WIPO (PCT)
Prior art keywords
sample data
labeling
labeling result
node
unknown sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2021/095786
Other languages
English (en)
French (fr)
Inventor
常庆龙
胡新宇
张彦芳
孙旭东
张亮
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP21837643.2A priority Critical patent/EP4163803A4/en
Publication of WO2022007527A1 publication Critical patent/WO2022007527A1/zh
Priority to US18/150,505 priority patent/US20230169096A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G06N20/20—Ensemble learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G06F16/285—Clustering or classification
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/088—Non-supervised learning, e.g. competitive learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G06N5/041—Abduction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/02—Knowledge representation; Symbolic representation
    • G06N5/022—Knowledge engineering; Knowledge acquisition
    • G06N5/025—Extracting rules from data

Definitions

  • the present application relates to the field of computer technology, and in particular, to a sample data labeling system, method and related equipment.
  • AI artificial intelligence
  • Most of the existing AI technologies including supervised learning (supervised learning), deep learning (DL), etc., are based on a large number of labeled data sets.
  • the data generates and/or adds labels capable of representing the classification of the sample data.
  • An existing general labeling method is to label the sample data in the central node set: each edge client node (referred to as the edge node) uploads the sample features that need to be labeled (including sample data or key features of the sample data) to the central node. , marked by the central node.
  • Another labeling method is to label the sample data at the edge nodes, that is, each edge node directly performs labeling operations locally after collecting the sample features to be labelled.
  • edge nodes such as personal photos stored on personal mobile phone terminals, Hospital patient data, etc.
  • uploading all the sample features of the edge nodes to the central node requires a large amount of data communication, resulting in high communication load. It needs to be completed at the central node, and the central node is also prone to form a bottleneck.
  • the latter is limited by the low computing power of edge nodes, and its label labeling efficiency is not high.
  • certain sample features cannot be labelled or labelled incorrectly. .
  • This application provides a sample data labeling system, method and related equipment, which are used to improve the labeling efficiency and labeling quality of AI sample data.
  • the present invention provides a sample data labeling system, including an edge node and a center node, the center node is connected with the edge node; the edge node is used for: acquiring key features of sample data; and judging samples according to the key features Whether the data is unknown sample data; when the sample data is unknown sample data, perform labeling processing on the sample data to obtain the first labeling result; upload the first labeling result to the central node.
  • the central node is used to: receive the first labeling result sent by the edge node; perform consistency processing on the first labeling result indicating that the unknown sample is successfully labelled to obtain the second labeling result; The result is subjected to labeling processing to obtain a third labeling result.
  • the labeling of the sample data by the edge nodes can improve the labeling efficiency; in addition, due to the objective conditions such as the low computing power of the edge nodes, the first labeling result obtained after labeling the edge nodes may have labeling errors or labeling errors. In the case of failure, etc., after the first labeling result is sent to the central node, the central node can correct the first labeling result or label it again, which can improve the quality of the labeling.
  • the edge node determines whether the sample data is unknown sample data according to the unknown sample model and key features obtained from the sample data.
  • the unknown sample model is generated according to a plurality of key features obtained from a known annotation result set.
  • the known labeling result set includes labeling results obtained by the central node before generating the unknown sample model. These labeling results include the second labeling result generated by the central node through consistency processing and/or the third labeling result obtained after labeling by the central node.
  • the unknown sample model may further include a multi-dimensional coordinate space, and the multi-dimensional coordinate space is generated by the central node according to the key features of the known labeling result set.
  • the coordinate distance is obtained, and according to the correlation indicated by the coordinate distance, it is determined whether the sample data is unknown sample data. If the coordinate distance is larger, the correlation is lower, and when the correlation is higher than a preset threshold, the sample data is determined as unknown sample data. If the coordinate distance is lower, the correlation is lower, and when the correlation is lower than a preset threshold, the sample data is determined as unknown sample data.
  • the unknown sample model may further include a neural network model. An inference action is performed on the sample data through the neural network model, and if a correct inference result cannot be obtained, the sample data is considered to be unknown sample data.
  • the first labeling result when the first labeling result indicates that the unknown sample data is successfully labelled, the first labeling result includes: a sample identifier of the sample data, a sample feature of the sample data, and the The edge node is a label determined by the sample data; when the first labeling result indicates that labeling the unknown sample data fails, the first labeling result includes: a sample identifier of the sample data and a sample feature of the sample data.
  • the sample features of the first annotation result include the sample data and/or the key features.
  • the central node when the consistency processing is performed on the first labeling result, performs clustering on a plurality of first labeling results including the first labeling result through similarity division, to obtain the group corresponding to the first labeling result. Further, in an optional implementation manner, the central node clusters a plurality of first labeling results including the first labeling results in an unsupervised manner, and the unsupervised manner includes one of the following or more: K-MEANS, KNN.
  • the central node when the labels of the labeling results in the group are inconsistent, performs an integrated decision on the labels of all labeling results in the group to obtain a group label. According to the group label The second labeling result is obtained.
  • the integrated decision includes: voting method or weighted voting method.
  • the central node generates a new unknown sample model or updates the unknown sample data model according to the second labeling result and/or the third labeling result.
  • the present application discloses a sample data labeling method, which is applied to an edge node of a sample labeling system.
  • the method includes: acquiring key features of sample data; judging whether the sample data is unknown sample data according to the key features; when the sample data is unknown sample data, labeling the unknown sample data to obtain the first A labeling result; sending the first labeling result to the central node.
  • the edge nodes only label the sample data that needs to be labeled, which can reduce the number of sample data that needs to be labeled, and send the labeled first labeling result to the central node for further processing, which can improve the labeling. quality.
  • the edge node determines whether the sample data is unknown sample data according to the unknown sample model and the key feature.
  • the unknown sample model is generated according to a plurality of key features obtained from a known annotation result set.
  • the known labeling result set includes successfully labeling results obtained by the central node before generating the unknown sample model.
  • the unknown sample model may further include a multi-dimensional coordinate space, and the multi-dimensional coordinate space is generated by the central node of the sample data labeling system according to the key features of the known labeling result set.
  • the coordinate distance is obtained, and according to the correlation indicated by the coordinate distance, it is determined whether the sample data is unknown sample data. If the coordinate distance is larger, the correlation is lower, and when the correlation is higher than a preset threshold, the sample data is determined as unknown sample data. If the coordinate distance is lower, the correlation is lower, and when the correlation is lower than a preset threshold, the sample data is determined as unknown sample data.
  • the unknown sample model may further include a neural network model. An inference action is performed on the sample data through the neural network model. If a correct inference result cannot be obtained, the sample data is considered to be unknown sample data.
  • the first labeling result when the first labeling result indicates that the unknown sample data is successfully labelled, the first labeling result includes: a sample identifier of the sample data, a sample feature of the sample data, and all The edge node is the label determined by the sample data; when the first labeling result indicates that labeling the unknown sample data fails, the first labeling result includes: the sample identifier of the sample data and the sample data sample features.
  • the sample features of the first annotation result include the sample data and/or the key features.
  • the present application discloses a sample data labeling method, which is used for a central node of a sample labeling system.
  • the method includes: receiving a first labeling result sent by an edge node in the sample labeling system, where the first labeling result is obtained by labeling unknown sample data by the edge node; when the first labeling result is When indicating that the labeling of the unknown sample data is successful, the consistency processing is performed on the first labeling result to obtain a second labeling result; when the labeling result indicates that the labeling of the unknown sample data fails, the unknown sample is labelled.
  • the data is labeled and processed to obtain a third labeling result.
  • the central node performs secondary processing on the first labeling result sent by the edge nodes, thereby improving the labeling quality of the sample data.
  • the central node when the consistency processing is performed on the first labeling result, performs clustering on a plurality of first labeling results including the first labeling result through similarity division , to obtain the group corresponding to the first labeling result. Further, in an optional implementation manner, the central node clusters a plurality of first labeling results including the first labeling results in an unsupervised manner, and the unsupervised manner includes one of the following or Various: K-MEANS, KNN.
  • the central node when the labels of the labeling results in the group are inconsistent, performs an integrated decision on the labels of all labeling results in the group to obtain a group label. According to the group label The second labeling result is obtained.
  • the integrated decision includes: voting method or weighted voting method.
  • the central node generates a new unknown sample model or updates the unknown sample data model according to the second labeling result and/or the third labeling result.
  • the present application discloses a node, which includes a functional module that executes the sample labeling method provided by the second aspect or any possible design of the second aspect;
  • the process steps of the sample labeling method in the second aspect are correspondingly divided into functional modules, and functional modules can also be divided according to specific implementation needs.
  • the present application discloses a node, which includes a functional module for executing the sample labeling method provided by the third aspect or any possible design of the third aspect;
  • the process steps of the sample labeling method in the second aspect are correspondingly divided into functional modules, and functional modules can also be divided according to specific implementation needs.
  • the present application discloses a computer program code.
  • the instructions included in the program code are executed by a computer, the computer can implement all of the above-mentioned second aspect or any possible implementation manner of the second aspect. Describe the sample data labeling method.
  • the present application discloses a computer program code.
  • the instructions included in the program code are executed by a computer, the computer can implement the third aspect or any of the possible implementations of the third aspect. Describe the sample data labeling method.
  • the present application discloses a computer-readable storage medium, where computer program instructions are stored in the computer-readable storage medium, and when the computer program instructions are executed on a computer, the computer is made to execute the above-mentioned second aspect Or the sample data labeling method in any possible implementation manner of the second aspect.
  • the present application discloses a computer-readable storage medium, where computer program instructions are stored in the computer-readable storage medium, and when the computer program instructions are executed on a computer, the computer is made to execute the above-mentioned third aspect Or the sample data labeling method in any possible implementation manner of the third aspect.
  • FIG. 1 is a schematic structural diagram of a sample data labeling system according to an embodiment of the present invention.
  • FIG. 2 is a schematic structural diagram of a central node or an edge node of a sample data labeling system according to an embodiment of the present invention
  • FIG. 3 is a schematic flowchart of a method for labeling sample data according to an embodiment of the present invention.
  • FIG. 4 is a schematic diagram of a format of an annotation result provided by an embodiment of the present invention.
  • 5A is a schematic diagram of a module of an edge node according to an embodiment of the present invention.
  • 5B is a schematic diagram of a module of a central node according to an embodiment of the present invention.
  • FIG. 6 is a schematic diagram of a multi-dimensional coordinate space provided by an embodiment of the present invention.
  • words such as “exemplary” or “for example” are used to represent examples, illustrations or illustrations. Any embodiments or designs described in the embodiments of the present application as “exemplary” or “such as” should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as “exemplary” or “such as” is intended to present the related concepts in a specific manner.
  • the meaning of "plurality” refers to two or more. For example, a plurality of nodes refers to two or more nodes. “At least one” refers to any number, eg, one, two, or more.
  • a and/or B can be A only, B only, or both A and B.
  • At least one of A, B, and C may be A only, B only, C only, or both A and B, B and C, A and C, or A, B and C included.
  • the terms “first” and “second” in this application are only used to distinguish different objects, and are not used to indicate the priority or importance of the objects.
  • a sample data labeling system 100 including a central node 110 and a plurality of edge nodes 120 .
  • the central node 110 communicates with the central node 110 through the bus 130 .
  • the edge nodes 120A, 120B, 120C, and 120D are connected to each other, and the network interface may be an interface connected to an internal network or an interface connected to the Internet. That is, through the bus 130, the labeling system can be deployed on the internal network (campus network, VPN, Cloud, etc.) or on the public network.
  • the central node 110 or the edge node 120 may be deployed on at least one processor platform 200 as shown in FIG. 2, which may be, for example, a server, a personal computer, a mobile device (eg, a phone, a smart phone, a tablet such as an iPad) computer), personal digital assistant (PDA), internet device, embedded device or any other type of computing device.
  • the processor platform 200 includes a memory 210, a processor 250, a bus 260, an interface device 270, and an I/O interface.
  • the memory 210 is used to store program logic 220 (eg, software) and data 230
  • the program logic 220 includes various logic modules for the labeling system 100 to perform functions such as consistency processing and labeling.
  • Data 230 may include sample data, annotation results, unknown sample data models, and the like.
  • the memory 210 may also contain an operating system 240 .
  • the processor 250 may include multiple processor cores (cores), and may also be a VLSI.
  • the processor 250 may use the operating system 240 to execute program logic 220 and process data 230 .
  • Bus 260 is used for communication between memory 210 , processor 250 , interface device 270 and I/O interface 280 .
  • a user may interact with the processor platform 200 through an interface device 270, such as a keyboard, mouse, or the like.
  • Processor platform 200 may also communicate with one or more other processor platforms through I/O interfaces, such as networks, modems, and the like.
  • the various components of the processor platform 200 provided in the embodiments of the present application are only exemplary, and those skilled in the art can add or reduce components as required, or divide the function of one component to implement multiple components.
  • the sample data labeling system 100 can start one or more sample data labeling tasks, and each sample data labeling task can be timed (for example, a set time or a set period) or according to an external trigger condition (such as the raw sample data reaches a threshold value). etc.) or start by command.
  • each sample data labeling task can be timed (for example, a set time or a set period) or according to an external trigger condition (such as the raw sample data reaches a threshold value). etc.) or start by command.
  • the following describes the process of labeling the sample data by the sample data labeling system 100 according to FIG. 3 . Unless otherwise specified, this process is applicable to any labeling task of the sample data labeling system 100 .
  • the sample data labeling method provided by the embodiment of the present application includes steps S311 - S314 and S321 - S323 .
  • the steps in FIG. 3 can be modified or deleted as needed, and the sequence of any steps in FIG. 3 can be adjusted as needed.
  • Steps S311 to S314 disclose a sample data labeling method applied in the edge node 310 .
  • the edge node 310 acquires key features of the sample data.
  • Obtaining the key feature of the sample data may be that the edge node 310 extracts the key feature from the sample data after obtaining the sample data, or may be the key feature that receives sample data sent by other devices.
  • Step 311 may be triggered when the edge node 310 receives the sample data, or may be triggered periodically.
  • the key feature is the relevant attributes of the sample data, such as one or more of the total length of the sample data, the average packet interval of the sample data, the upstream and downstream directions, the start time and/or the end period of the sample data collection.
  • the edge node 310 determines whether the sample data is unknown sample data according to the acquired key features. In an implementation manner, the edge node 310 maps the key features obtained in step S311 to the unknown sample data model of the edge node, so as to determine whether the sample data is unknown sample data. In one implementation, the unknown sample data model is generated by the central node 320 according to a plurality of key features obtained from the known annotation result set.
  • the known labeling result set includes a plurality of labeling results obtained by the central node 320 before generating the unknown sample data model, and each labeling result includes the sample identifier of the sample data, the sample feature of the sample data and the The label, the label may be marked by the central node 320, or may be marked by the edge node 310 and obtained after consistency processing by the central node.
  • the central node 320 when the sample data labeling system 100 performs labeling for the first time, the central node 320 generates a known labeling result set according to an empty labeling result set or an externally loaded labeling result set.
  • the unknown sample data model includes a multi-dimensional coordinate space
  • the multi-dimensional coordinate space is generated by the central node 320 according to the key features of the known labeling result set.
  • the multi-dimensional coordinate space 640 is a three-dimensional coordinate space composed of key features 601 to 603.
  • the key features 601 to 603 may be the key features of the sample data or the features mapped by the key features of the sample data. This key feature or mapped feature can be one or more.
  • the sample data labeling system 100 can map ten key features of the sample data into one three-dimensional coordinate space, and can also map the ten key features of the sample data into another ten-dimensional coordinate space.
  • the correlation between the sample data can be represented by the distance of the coordinates, and when the correlation is lower than a preset threshold, the sample data is determined as unknown sample data.
  • the edge node 310 may preset one or more thresholds. If the coordinate distance is larger, the correlation is lower.
  • the specific implementation of step S312 may be, for example, the key features of the sample data obtained in step S310, or according to this For the mapping feature of the key feature, when the coordinate distance in the multi-dimensional coordinate space is greater than or equal to a preset threshold, it can be determined that the sample data is unknown sample data.
  • step S312 may be, for example, the key feature of the sample data obtained in step S310, or the mapping feature according to the key feature, in the multi-dimensional coordinate space.
  • the coordinate distance is less than or equal to the preset threshold, it can be determined that the sample data is unknown sample data.
  • 611 to 613 are known labels, and the known labels are labels of the sample data of the known labeling result set.
  • the coordinate distances from the known labels 611-613 are generated according to their multi-dimensional coordinates.
  • the greater the coordinate distance The lower the correlation of the big marks the coordinate distances of 621 and 622 are lower than the preset threshold, and 621 and 622 are determined to be known sample data, while the coordinate distances of 631 to 634 are higher than the preset threshold, and 631 to 634 are determined to be unknown. sample.
  • the unknown sample data model further includes a neural network model.
  • An inference action can be performed on the key features obtained in step S311 through the neural network model.
  • the neural network model cannot obtain a correct inference result, the sample data is considered to be unknown sample data.
  • the edge node 310 annotates the sample data to obtain a first annotation result.
  • the format of the labeling result in this application is shown in FIG. 4 , including a sample identifier 401 and/or a sample feature 402 , and in some embodiments, the labeling result further includes a label 403 .
  • the sample identifier 401 is obtained according to the identifier of the sample data, which may be the same as the identifier of the sample data, or may be other identifiers generated from the identifier of the sample data.
  • the sample features 402 may include one or more key features of the sample data, and/or the sample data itself.
  • the label 403 is a label obtained after the edge node 310 labels the unknown sample data, and the label 403 may be one or more.
  • the sample data labeling system 100 uses the label value of the first labeling result to indicate whether the unknown sample data is labelled successfully in step S313.
  • the first labeling result includes the labeled label.
  • the labeling in step S313 is unsuccessful, the first labeling result does not include a label or the label value is empty.
  • the sample data labeling system 100 can also use a special parameter to indicate whether the unknown sample data is successfully labelled in step S313.
  • step S314 the edge node 310 sends the first labeling result to the central node 320.
  • Steps S321 to S323 disclose a sample data labeling method applied to the central node 320. After the central node 320 receives the first labeling result sent by each edge node (120A, 120B, 120C, 120D in FIG. In S321, it is determined whether the received first labeling result indicates that the labeling is successful. In an implementation manner, step S321 may be triggered after the central node 320 receives a certain number of first labeling results from each edge node, or triggered periodically or in any other manner. Further, in an implementation manner, step S321 will judge the first labeling results received by all the central nodes 320, and divide all the first labeling results into the first labeling results indicated as successful labeling and the first labeling results indicated as labeling failures.
  • step S322 and step S323 After the two categories of results are marked, the operations of step S322 and step S323 are performed respectively. Further, when step S321 judges that the received first labeling results are all indicated as labeling success, the central node 320 will not execute step S323, or when S321 judges that the received first labeling results are all indicated as labeling failure, the central node 320 will not perform step S323. Node 320 will not execute step S322.
  • the central node 320 may also perform batch processing on the received first annotation results, that is, the first annotation results are divided into several batches, and S321 to S323 are executed for the first batch of first annotation results. After the steps of S321-S323 are performed again for the first labeling results of the second batch, and so on, until all the first labeling results received are processed.
  • step S322 When step S321 indicates that the first labeling result is successfully labelled, step S322 is entered. In step S322, the central node 320 obtains the second labeling result after performing consistency processing on the first labeling result indicating that the labeling is successful.
  • the format of the second annotation result is shown in Figure 4. Through similarity division, the central node 320 clusters a plurality of first labeling results including the first labeling result, that is, using a clustering algorithm to classify a plurality of first labeling results that are successfully labelled into different groups, belonging to the same
  • the first annotation results of a group contain the same or similar sample features, while the first annotation results belonging to different groups have distinct sample characteristics.
  • clustering can be performed in an unsupervised manner, and the unsupervised manner includes K-MEANS (k-means clustering algorithm), KNN (k-nearest neighbors algorithm), and the like.
  • K-MEANS k-means clustering algorithm
  • KNN k-nearest neighbors algorithm
  • the central node 320 performs consistency processing on all labeling results in the group, that is, through the integrated decision-making process.
  • the group label of the group is determined, and the group label is determined as the new label of each first labeling result in the group, and the second labeling result corresponding to each first labeling result is obtained.
  • the central node 320 When the group label is the same as the label of a first labeling result, the central node 320 maintains the first labeling result, in this case, it can be considered that the second labeling result corresponding to the first labeling result is the same as the first labeling result; When the group label is different from the label of a first labeling result, the central node 320 modifies the label of the first labeling result to a group label. In this case, the second labeling result corresponding to the first labeling result is given to the label of the first labeling result.
  • the first annotation results are different.
  • the integrated decision includes voting or weighted voting. Tag is assumed to be determined are ⁇ C 1, C 2, ...
  • T edges annotation system 100 sample data node 120 of the sample data x are pre-annotation result (h 1 (x), h 2 (x)...h T (x)).
  • the label C i with the largest number is the final label, that is, the group label obtained by the integrated decision. If more than one label gets the most votes, one is randomly selected to be the group label.
  • the voting method can also be a supermajority voting method, that is, a majority of the votes. On the basis of the relative majority voting method, in addition to obtaining the highest number of votes, a majority of the votes are also required.
  • the weighted voting method is that the label votes of T edge nodes 120 are multiplied by a weight, and finally the weighted votes of the same label are summed up, and the label corresponding to the largest value is the final label, that is, the group label obtained by the integrated decision.
  • step S323 is entered.
  • the central node 320 labels the first labeling result indicated as labeling failure to obtain a third labeling result.
  • the format of the third labeling result is shown in FIG. 4 .
  • the edge nodes and/or central nodes of the sample data labeling system 100 label the sample data by means of rule matching, rule association or manual labeling. Rule matching is to mark the sample data whose characteristics of the sample data match the rule by setting a certain rule. For example, if the keyword YouTube is found in the domain name system (DNS) of the network flow, it can be determined that the network flow is video stream.
  • DNS domain name system
  • Rule association refers to determining the same label for multiple sample data that are related to each other. If an unknown network flow always appears in association with a known network flow, it is determined that the unknown network flow is a network of the same category as the known network flow. flow. Manual labeling is to manually observe the sample features and label the results.
  • the central node 320 after the central node 320 obtains the second and third annotation results, it can generate a new unknown sample model or update the original unknown sample model according to the second and third annotation results.
  • the newly generated or updated unknown sample model is sent to the edge node 310, and the edge node 310 updates the locally existing unknown sample model after receiving the unknown sample model.
  • the edge node 310 or the central node 320 may perform encryption processing on the sent data, and perform decryption processing on the received data, where the encrypted/decrypted data includes the first annotation result or Unknown sample model.
  • the edge node or the central node in the embodiments of the present application includes corresponding hardware structures and/or software modules for executing each function.
  • the functions and steps of each example described in the embodiments disclosed in this application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is performed by hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods, but such implementations should not be considered beyond the scope of this application.
  • the embodiment of the present application provides two kinds of nodes 500 and 510 .
  • the node 500 is used as an edge node of the sample data labeling system 100 , and the node 500 includes an acquisition unit 501 , an identification unit 502 , a labeling unit 503 and a communication unit 504 .
  • the acquiring unit 501 is configured to execute step S311
  • the identifying unit 502 is configured to execute step S312
  • the labeling unit 503 is configured to execute step S313
  • the communication unit 504 is configured to execute step S314.
  • the node 510 is used as the central node of the sample data labeling system 100 , and the node 510 includes a communication unit 511 , a processing unit 512 and an labeling unit 513 .
  • the communication unit 511 is configured to receive the first labeling result sent by the edge node 310
  • the processing unit 512 is configured to execute step S322
  • the labeling unit 513 is configured to execute step S323.
  • the constituent units of the node 500 or 510 provided in this embodiment of the present application are only exemplary, and those skilled in the art may increase or decrease the constituent units as required, or divide the function of one unit into multiple units to implement.
  • Embodiments of the present application also provide a computer-readable storage medium for storing program codes for implementing the foregoing method for labeling sample data, where the program codes include instructions for executing the method process described in any one of the foregoing method embodiments.
  • the aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk, optical disk, random access memory (RAM), solid state disk (SSD) or non-volatile memory (non-volatile memory) and other various storage media.
  • a non-transitory machine-readable medium that can store program code.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computational Linguistics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Image Analysis (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

一种样本数据标注系统，应用于该样本数据标注系统的标注方法和相关装置。该样本数据标注系统包括边缘节点和中心节点，该边缘节点获取样本数据的关键特征（S311），根据所述关键特征判断样本数据是否为未知样本数据（S312），当所述样本数据为未知样本数据时，对所述样本数据进行标注处理，得到第一标注结果（S313），将所述第一标注结果发送至中心节点（S314）；中心节点接收所述第一标注结果，判断第一标注结果是否指示为标注成功（S321），当所述第一标注结果指示对所述未知样本数据标注成功时，对所述第一标注结果进行一致性处理，得到第二标注结果（S322），当所述标注结果指示对所述未知样本数据标注失败时，对所述未知样本数据进行标注处理，得到第三标注结果（S323）。

Description

样本数据标注系统、方法以及相关设备
本申请要求于2020年07月06日提交的申请号为202010642572.5、申请名称为“样本数据标注系统、方法以及相关设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及计算机技术领域，尤其涉及一种样本数据标注系统、方法及相关设备。
背景技术
近来，人工智能(artificial intelligence，AI)在越来越多的领域得到了应用。现有的绝大部分AI技术，包括监督学习(supervised learning)、深度学习(deep learning，DL)等，都是建立在大量被标注的数据集的基础上的，其中，标注，是指为样本数据生成和/或添加能够表示该样本数据的分类的标签。现有通用的一种标注方法是在中心节点集中对样本数据进行标注：各边缘客户端节点(简称边缘节点)将需要标注的样本特征(包括样本数据或样本数据的关键特征)上传至中心节点，由中心节点进行标注。另一种标注方法是在边缘节点对样本数据进行标注，即各边缘节点在采集完所需标注的样本特征后，直接在其本地各自进行标注操作。这两种标注方式都有一定的缺陷，前者一方面受限于隐私保护的需要，如在联邦学习(federated learning,FL)领域,边缘节点的样本数据(如个人手机终端上存储的个人照片、医院的病人资料等)不能上传至中心节点进行标注，另一方面，将边缘节点的所有样本特征上传至中心节点，需要很大的数据通信量，造成通信负载过高，另外由于所有的标注都需要在中心节点完成，中心节点也容易形成瓶颈。后者由于受边缘节点的计算力低的条件限制，其标签标注效率不高，此外，由于其标签标注方法相较于中心标注往往更简易，会造成一定的样本特征无法标注或标注错误的情况。
因此，如何提升样本数据标注的效率和标注质量就成为AI领域的一个研究问题。
发明内容
本申请提供了一种样本数据标注系统、方法和相关设备，用于提升AI样本数据的标注效率和标注质量。
第一方面，本发明提供了一种样本数据标注系统，包括边缘节点和中心节点，该中心节点与边缘节点相连接；该边缘节点用于：获取样本数据的关键特征；根据关键特征，判断样本数据是否为未知样本数据；当样本数据为未知样本数据时，对样本数据进行标注处理，得到第一标注结果；将第一标注结果上传至中心节点。该中心节点用于：接收边缘节点发送的第一标注结果；对指示为未知样本标注成功的第一标注结果 进行一致性处理，得到第二标注结果；对指示为未知样本标注失败的第一标注结果进行标注处理，得到第三标注结果。
本申请中，由边缘节点对样本数据进行标注，可以提高标注的效率；此外，由于边缘节点的计算能力比较低等客观条件，边缘节点标注后得到的第一标注结果可能存在标注错误，或标注失败等情况，将第一标注结果发送至中心节点后，该中心节点可以对第一标注结果进行纠正或再次标注，可以提高标注的质量。
在一种可选的实施方式中，该边缘节点根据未知样本模型和从样本数据中获取到的关键特征判断所述样本数据是否为未知样本数据。
在一种可选的实施方式中，该未知样本模型是根据从已知标注结果集中获取的多个关键特征生成的。该已知标注结果集包括所述中心节点在生成所述未知样本模型前获取到的标注结果。这些标注结果包含该中心节点通过一致性处理后生成的第二标注结果和/或中心节点标注后得到的第三标注结果。
在一种可选的实施方式中，该未知样本模型还可以包括多维坐标空间，该多维坐标空间是所述中心节点根据已知标注结果集的关键特征所生成的。根据所述样本数据的关键特征或所述关键特征的映射特征，得到坐标距离，根据所述坐标距离表示的相关性，确定所述样本数据是否为未知样本数据。如果所述坐标距离越大，表示相关性越低，当所述相关性高于预设的阈值时，将所述样本数据确定为未知样本数据。如果所述坐标距离越低，表示相关性越低，当所述相关性低于预设的阈值时，将所述样本数据确定为未知样本数据。
在一种可选的实施方式中，所述未知样本模型还可以包括神经网络模型。通过该神经网络模型对样本数据进行推理(Inference)动作，若无法得到正确的推理结果，则认为该样本数据为未知样本数据。
在一种可选的实施方式中，当第一标注结果指示对该未知样本数据标注成功时，该第一标注结果包括：所述样本数据的样本标识、所述样本数据的样本特征和所述边缘节点为所述样本数据确定的标签；当该第一标注结果指示对该未知样本数据标注失败时，该第一标注结果包括：所述样本数据的样本标识和所述样本数据的样本特征。在一种可选的实施方式中，第一标注结果的样本特征包括所述样本数据和/或所述关键特征。
在一种可选的实施方式中，当对第一标注结果进行一致性处理时，该中心节点通过相似度划分，将包括所述第一标注结果的多个第一标注结果进行聚类，得到所述第一标注结果对应的组。进一步地，在一种可选的实施方式中，该中心节点通过非监督的方式将包括所述第一标注结果的多个第一标注结果进行聚类，，所述非监督方式包括如下一种或多种：K-MEANS、KNN。
在一种可选的实施方式中，当所述组中的各标注结果的标签不一致时，该中心节点对所述组中的所有标注结果的标签执行集成决策得到组标签，根据所述组标签得到所述第二标注结果。在一种可选的实施方式中，所述集成决策包括：投票法或加权投票法。
在一种可选的实施方式中，该中心节点根据所述第二标注结果和/或第三标注结果生成新的未知样本模型或更新所述未知样本数据模型。
第二方面，本申请公开了一种样本数据标注方法，应用于样本标注系统的边缘节点。该方法包括：获取样本数据的关键特征；根据所述关键特征判断所述样本数据是否为未知样本数据；当所述样本数据为未知样本数据时，对所述未知样本数据进行标注处理，得到第一标注结果；将所述第一标注结果发送至中心节点。
通过该方法，边缘节点仅对所需要标注的样本数据进行标注，可以减少需要标注的样本数据的数量，并将标注后的第一标注结果，发送至中心节点，做进一步的处理，可以提高标注质量。
在一种可选的实施方式中，边缘节点根据未知样本模型和所述关键特征判断所述样本数据是否为未知样本数据。
在一种可选的实施方式中，未知样本模型是根据从已知标注结果集中获取的多个关键特征生成的。该已知标注结果集包括中心节点在生成所述未知样本模型前获取到的标注成功的标注结果。
在一种可选的实施方式中，该未知样本模型还可以包括多维坐标空间，该多维坐标空间是样本数据标注系统的中心节点根据已知标注结果集的关键特征所生成的。根据所述样本数据的关键特征或所述关键特征的映射特征，得到坐标距离，根据所述坐标距离表示的相关性，确定所述样本数据是否为未知样本数据。如果所述坐标距离越大，表示相关性越低，当所述相关性高于预设的阈值时，将所述样本数据确定为未知样本数据。如果所述坐标距离越低，表示相关性越低，当所述相关性低于预设的阈值时，将所述样本数据确定为未知样本数据。
在一种可选的实施方式中，所述未知样本模型还可以包括神经网络模型。通过该神经网络模型对样本数据进行推理(Inference)动作，若无法得到正确的推理结果，则认为该样本数据是未知样本数据。
在一种可选的实施方式中，当第一标注结果指示对所述未知样本数据标注成功时，该第一标注结果包括：所述样本数据的样本标识、所述样本数据的样本特征和所述边缘节点为所述样本数据确定的标签；当所述第一标注结果指示对所述未知样本数据标注失败时，所述第一标注结果包括：所述样本数据的样本标识和所述样本数据的样本特征。在一种可选的实施方式中，第一标注结果的样本特征包括所述样本数据和/或所述关键特征。
第三方面，本申请公开了一种样本数据标注方法，该方法用于样本标注系统的中心节点。该方法包括：接收所述样本标注系统中的边缘节点发送的第一标注结果,所述第一标注结果是由所述边缘节点对未知样本数据进行标注处理得到的；当所述第一标注结果指示对所述未知样本数据标注成功时，对所述第一标注结果进行一致性处理，得到第二标注结果；当所述标注结果指示对所述未知样本数据标注失败时，对所述未知样本数据进行标注处理，得到第三标注结果。
通过该方法，中心节点对边缘节点发送的第一标注结果进行二次处理，提高了样本数据的标注质量。
在一种可选的实施方式中，当对所述第一标注结果进行一致性处理时，该中心节点通过相似度划分，将包括所述第一标注结果的多个第一标注结果进行聚类，得到所述第一标注结果对应的组。进一步地，在一种可选的实施方式中，该中心节点通过非监 督的方式将包括所述第一标注结果的多个第一标注结果进行聚类，所述非监督方式包括如下一种或多种：K-MEANS、KNN。
在一种可选的实施方式中，当所述组中的各标注结果的标签不一致时，该中心节点对所述组中的所有标注结果的标签执行集成决策得到组标签，根据所述组标签得到所述第二标注结果。在一种可选的实施方式中，所述集成决策包括：投票法或加权投票法。
在一种可选的实施方式中，该中心节点根据所述第二标注结果和/或第三标注结果生成新的未知样本模型或更新所述未知样本数据模型。
第四方面，本申请公开了一种节点，该节点包括执行第二方面或第二方面的任意可能设计提供的样本标注方法的功能模块；本申请对功能模块的划分不做限定，可以按照第二方面的样本标注方法的流程步骤对应划分功能模块，也可以按照具体实现需要划分功能模块。
第五方面，本申请公开了一种节点，该节点包括执行第三方面或第三方面的任意可能设计提供的样本标注方法的功能模块；本申请对功能模块的划分不做限定，可以按照第二方面的样本标注方法的流程步骤对应划分功能模块，也可以按照具体实现需要划分功能模块。
本申请上述不同方面的各实施方式可以在不冲突的情况下互相结合或引用。
第六方面，本申请公开了一种计算机程序代码，所述程序代码包括的指令被计算机执行时，能够使所述计算机实现上述第二方面或第二方面任意一种可能的实现方式中的所述样本数据标注方法。
第七方面，本申请公开了一种计算机程序代码，所述程序代码包括的指令被计算机执行时，能够使所述计算机实现上述第三方面或第三方面任意一种可能的实现方式中的所述样本数据标注方法。
第八方面，本申请公开了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序指令，当所述计算机程序指令在计算机上运行时，使得所述计算机执行上述第二方面或第二方面任意一种可能的实现方式中的所述样本数据标注方法。
第九方面，本申请公开了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序指令，当所述计算机程序指令在计算机上运行时，使得所述计算机执行上述第三方面或第三方面任意一种可能的实现方式中的所述样本数据标注方法。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对实施例描述中所需要使用的附图做简单的介绍，显而易见，下面描述中的附图仅仅是本发明的一些实施例。
图1为本发明实施例提供的一种样本数据标注系统的结构示意图；
图2为本发明实施例提供的一种样本数据标注系统的中心节点或边缘节点的结构示意图；
图3为本发明实施例提供的一种样本数据标注方法的流程示意图；
图4为本发明实施例提供的一种标注结果的格式示意图；
图5A为本发明实施例提供的一种边缘节点的模块示意图；
图5B为本发明实施例提供的一种中心节点的模块示意图；
图6为本发明实施例提供的一种多维坐标空间示意图。
具体实施方式
为了使本技术领域的人员更好地理解本发明方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚的描述。显然，所描述的实施例仅仅是本发明一部分的实施例，而不是全部的实施例。
在本申请实施例中，“示例性的”或者“例如”等词用于表示作例子、例证或说明。本申请实施例中被描述为“示例性的”或者“例如”的任何实施例或设计方案不应被解释为比其它实施例或设计方案更优选或更具优势。确切而言，使用“示例性的”或者“例如”等词旨在以具体方式呈现相关概念。在本申请实施例中，除非另有说明，“多个”的含义是指两个或两个以上。例如，多个节点是指两个或两个以上的节点。“至少一个”是指任意的数量，例如，一个，两个或两个以上。“A和/或B”可以是只有A，只有B，或包括A和B。“A、B和C中的至少一个”，可以是只有A，只有B，只有C，或包括A和B，包括B和C，包括A和C，或者包括A，B和C。本申请中的“第一”、“第二”等用语仅用于区分不同的对象，而不用于对象的指示优先级或重要性。
为了提高样本数据标注的效率和质量，在一个实施方式中，如图1所示，提供了一种样本数据标注系统100，包括中心节点110和多个边缘节点120，中心节点110通过总线130与各边缘节点120A、120B、120C、120D相连接，该网络接口可以是连接内部网络的接口，也可以是连接Internet的接口。即通过总线130，该标注系统可以部署在内部网络(园区网络、VPN、Cloud等)，也可以部署在公网上。
中心节点110或边缘节点120可以部署在如图2所示的至少一个处理器平台200上，处理器平台200可以是如服务器、个人计算机、移动设备(例如，电话、智能电话、诸如iPad的平板电脑)、个人数字助理(PDA)、互联网设备、嵌入式设备或任何其他类型的计算设备。进一步地，处理器平台200包含存储器210、处理器250、总线260、接口设备270和I/O接口。存储器210用于存储程序逻辑220(如软件)和数据230，程序逻辑220包括标注系统100执行一致性处理、标注等功能的各逻辑模块。数据230可以包括样本数据、标注结果、未知样本数据模型等。存储器210还可以包含操作系统240。处理器250可以包括多个处理器核心(core),也可以是一块超大规模集成电路，处理器250可以利用操作系统240执行程序逻辑220，并对数据230进行处理。总线260用于存储器210、处理器250、接口设备270和I/O接口280之间的通信。用户可以通过接口设备270，如键盘、鼠标等与处理器平台200交互。处理器平台200还可以通过I/O接口，如网络、调制解调器等与其他的一个或多个处理器平台通信。本申请实施例提供的处理器平台200的各个组成部分仅仅是示例性的，本领域技术人员可以根据需要增加或减少组件，也可以把一个组件的功能分割由多个组件来实现。
样本数据标注系统100可以启动一次或多次样本数据标注任务，每一次的样本数据标注任务，可以定时(例如，设定时间或设定周期)或根据外部触发条件(如原始样本数据到达一个阈值等)或根据指令启动。以下根据图3来介绍样本数据标注系统100 进行样本数据标注的流程，除非另有说明，本流程适用于样本标注系统100的任何一次标注任务。
如图3所示，为本申请实施例提供的样本数据标注方法，该方法包括步骤S311～S314和S321～S323。在具体实现中，图3中的步骤可以根据需要进行修改或删除，并且图3中的任意步骤的顺序可以根据需要进行调整。
步骤S311～S314公开了应用在边缘节点310中的样本数据标注方法。在步骤S311中，边缘节点310获取样本数据的关键特征。获取样本数据的关键特征，可以是边缘节点310在获取样本数据后从该样本数据中提取该关键特征，也可以是接收其他设备发送的样本数据的关键特征。步骤311可以在边缘节点310接收到样本数据时触发，也可以定时触发。关键特征是样本数据的相关属性，如样本数据的总长度、样本数据的包平均间隔、上下行方向、该样本数据的采集起始时间和/或结束周期中的一个或多个。
在步骤S312中，边缘节点310根据已获取的关键特征，判断所述样本数据是否为未知样本数据。在一个实现方式中，边缘节点310将步骤S311所获取的关键特征映射到该边缘节点的未知样本数据模型，以判断所述样本数据是否为未知样本数据。在一个实现方式中，该未知样本数据模型是中心节点320根据从已知标注结果集中获取的多个关键特征所生成的。该已知标注结果集包含由中心节点320在生成未知样本数据模型前获取到的多个标注成功的标注结果，每个标注结果包括样本数据的样本标识、样本数据的样本特征和该样本数据的标签，该标签可以是由中心节点320标注的，也可以是由边缘节点310标注并由中心节点进行一致性处理后得到的。在另一个实现方式中，在样本数据标注系统100第一次进行标注时，中心节点320根据空的标注结果集或外部加载的标注结果集生成一个已知标注结果集。在一个实现方式中，未知样本数据模型包括多维坐标空间，该多维坐标空间是所述中心节点320根据已知标注结果集的关键特征所生成的。如图6所示，多维坐标空间640是由关键特征601～603所组成的三维坐标空间，关键特征601～603可以是样本数据的关键特征，也可以是样本数据的关键特征所映射的特征，这个关键特征或所映射的特征可以是一个或多个。在一个示例中，样本数据标注系统100可以将样本数据的十个关键特征映射成一个三维坐标空间，也可以将样本数据的十个关键特征映射成另一个十维坐标空间。
在样本数据标注系统100中，可以通过坐标距离的远近来表示样本数据之间的相关性，当相关性低于预设的阈值时，将样本数据确定为未知样本数据。边缘节点310可以预先设置一个或多个阈值，如果所述坐标距离越大表示相关性越低，步骤S312的具体实现方式可以是，如，步骤S310获取到的样本数据的关键特征，或依据此关键特征的映射特征，在所述多维坐标空间中的坐标距离，大于或等于预设的阈值时，则可确定该样本数据为未知样本数据。如果所述坐标距离越小表示相关性越低，步骤S312的具体实现可以是，如，步骤S310获取到的样本数据的关键特征，或依据此关键特征的映射特征，在所述多维坐标空间中的坐标距离，小于或等于预设的阈值时，则可确定该样本数据为未知样本数据。如图6所示，611～613是已知标签，该已知标签是已知标注结果集的样本数据的标签。当待标注样本数据(621～622、631～634)的关键特征映射至多维坐标空间640后，根据其多维坐标，生成与已知标签611～613的坐标距离， 在图6中，坐标距离越大标识相关性越低，621、622的坐标距离低于预设的阈值，确定621、622为已知样本数据，而631～634的坐标距离高于预设的阈值，确定631～634为未知样本数据。
在一个实现方式中，所述未知样本数据模型还包括神经网络模型。可以通过该神经网络模型对步骤S311获取到的关键特征做推理(Inference)动作，当该神经网络模型无法得到正确的推理结果时，则认为该样本数据是未知样本数据。
当在步骤S312中判断该样本数据为未知样本数据时，在步骤S313中，边缘节点310对样本数据进行标注，得到第一标注结果。在一个实现方式中，本申请中的标注结果的格式如图4所示，包括样本标识401和/或样本特征402，在一些实施例中，标注结果还包括标签403。样本标识401是根据样本数据的标识得到，可以跟样本数据的标识相同，也可以是跟样本数据的标识生成的其他标识。样本特征402可以包括样本数据的一个或多个关键特征，和/或，样本数据本身。标签403是边缘节点310对未知样本数据进行标注后得到的标签，标签403可以是一个或多个。在一个实现方式中，样本数据标注系统100使用第一标注结果的标签值，来指示步骤S313中对未知样本数据是否标注成功，当边缘节点310对未知样本数据标注成功时，第一标注结果包括所标注的标签。当步骤S313标注不成功时，则第一标注结果不包括标签或标签值为空。进一步地，样本数据标注系统100还可以用一个专有的参数来指示步骤S313对未知样本数据是否标注成功，如当该参数的值为True时，指示未知样本数据标注成功，当值为False时，指示未知样本数据标注失败。在步骤S314中，边缘节点310将第一标注结果发送至中心节点320。
步骤S321～S323公开了应用在中心节点320的样本数据标注方法，该中心节点320在接收了各边缘节点(如图1的120A、120B、120C、120D)发送的第一标注结果后，在步骤S321中判断所接收到的第一标注结果是否指示标注成功。在一个实现方式中，步骤S321可以在中心节点320接收各边缘节点一定数量的第一标注结果后被触发，或被定时触发或被其他任意的方式触发。进一步地，在一个实现方式中，步骤S321会判断所有中心节点320接收到的第一标注结果，将所有第一标注结果分为指示为标注成功的第一标注结果和指示为标注失败的第一标注结果两大类后，分别进行步骤S322和步骤S323的操作处理。进一步地，当步骤S321判断接收到的第一标注结果都指示为标注成功时，中心节点320不会执行步骤S323，或当后S321判断接收到的第一标注结果都指示为标注失败时，中心节点320不会执行步骤S322。
在另一个实现方式中，中心节点320还可以对接收到的第一标注结果进行分批处理，即将第一标注结果分成几个批次，第一批次的第一标注结果执行完S321～S323的步骤后，第二批次的第一标注结果再次执行S321-S323步骤，以此类推，直到所有接收到的第一标注结果的被处理完。
当步骤S321指示第一标注结果标注成功时，进入步骤S322。在步骤S322中，中心节点320对所述指示为标注成功的第一标注结果，进行一致性处理后，得到第二标注结果。第二标注结果的格式如图4所示。通过相似度划分，中心节点320将包括该第一标注结果的多个第一标注结果进行聚类，即用聚类算法将标注成功的多个第一标注结果分到不同的组中，属于同一组的第一标注结果含有相同或相似的样本特征，而 属于不同组的第一标注结果有着区别明显的样本特征。在一个实现方式中，可以通过非监督的方式进行聚类，所述非监督方式包括K-MEANS(k-means clustering algorithm)、KNN(k-nearest neighbors algorithm)等。以第一标注结果A为例，当第一标注结果A对应的组中各第一标注结果的标签不一致时，中心节点320对该组中的所有标注结果进行一致性处理，即通过集成决策的方式，决策出该组的组标签，并将组标签确定为该组中的每个第一标注结果的新标签，得到每个第一标注结果对应的第二标注结果。当组标签与一个第一标注结果的标签相同时，中心节点320维持该第一标注结果，这种情况下，可以认为该第一标注结果对应的第二标注结果与该第一标注结果相同；当组标签与一个第一标注结果的标签不相同时，中心节点320将该第一标注结果的标签修改为组标签，这种情况下，该第一标注结果对应的第二标注结果给与该第一标注结果不同。在一个实现方式中，该集成决策包括投票法或加权投票法。假设待确定的标签分别是{C 1,C 2,...C k},样本数据标注系统100的T个边缘节点120对样本数据x的预标注结果分别是(h 1(x),h 2(x)…h T(x))。通过相对多数投票法，也就是常说的少数服从多数，T个边缘节点120对样本数据x的标注结果中，数量最多的标签C i为最终确定的标签，也就是集成决策得到的组标签。如果不止一个标签获得最高票，则随机选择一个做组标签。投票法还可以是绝对多数投票法，也就是票数要过半数。在相对多数投票法的基础上，除了票数要获得最高票，还要求票数要过半数。加权投票法是T个边缘节点120的标签票数要乘以一个权重，最终将同一标签的加权票数求和，最大的值对应的标签为最终标签，也就是集成决策得到的组标签。
当步骤S321指示第一标注结果标注失败时，进入步骤S323。在步骤S323中，中心节点320对指示为标注失败的第一标注结果进行标注，得到第三标注结果，第三标注结果的格式如图4所示。在一个实现方式中，样本数据标注系统100的边缘节点和/或中心节点采用规则匹配、规则关联或人工标注的方式为样本数据标注标签。规则匹配是通过设定一定的规则，对样本数据特征匹配该规则的样本数据进行标注，如网络流的域名系统(domain name system，DNS)中发现了关键词YouTube，则可确定该网络流为视频流。规则关联是指为相互关联的多个样本数据确定相同的标签，如某未知网络流总是与某已知网络流关联出现，则确定此未知网络流为与已知网络流为同一类别的网络流。人工标注是通过人工观察样本特征并标注结果。
进一步地，在一个实现方式中，当中心节点320得到第二、第三标注结果后，可以根据所述第二、第三标注结果，生成新的未知样本模型或更新原有的未知样本模型。并将新生成的或者更新后的未知样本模型发送至边缘节点310，边缘节点310收到该未知样本模型后，更新本地已有的未知样本模型。
进一步地，在一个实现方式中，边缘节点310或中心节点320可以对发送的数据进行加密处理，并对接收到的数据进行解密处理，所述加/解密的数据包括所述第一标注结果或未知样本模型。
上面从系统的角度对本申请实施例提供的样本数据标注方法进行了介绍。可以理解的是，本申请实施例中的边缘节点或中心节点为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，本申请中所公开的实施例描述的各示例的功能和步骤，能够以硬件或硬件和计算机软件的结合等 形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
以下从不同的角度对本申请的节点的结构予以说明。为了实现本申请图3所示的方法，本申请实施例提供了了两种节点500、510。如图5A所示，节点500用于样本数据标注系统100的边缘节点，节点500包括获取单元501、识别单元502、标注单元503和通信单元504。本申请实施例中，获取单元501用于执行步骤S311，识别单元502用于执行步骤S312，标注单元503用于执行步骤S313，通信单元504用于执行步骤S314。如图5B所示，节点510用于样本数据标注系统100的中心节点，节点510包括通信单元511、处理单元512和标注单元513。通信单元511用于接收边缘节点310发送的第一标注结果，处理单元512用于执行步骤S322，标注单元513用于执行步骤S323。本申请实施例提供的节点500或510的各个组成单元仅仅是示例性的，本领域技术人员可以根据需要增加或减少组成单元，也可以把一个单元的功能分割由多个单元来实现。
本申请实施例也提供了计算机可读存储介质，用于存储实现上述样本数据标注方法的程序代码，所述程序代码包括的指令用于执行前述任意一个方法实施例所述的方法流程。前述的存储介质包括：U盘、移动硬盘、磁碟、光盘、随机存储器(random access memory，RAM)、固态硬盘(solid state disk，SSD)或者非易失性存储器(non-volatile memory)等各种可以存储程序代码的非短暂性的(non-transitory)机器可读介质。
需要说明的是，本申请所提供的实施例仅仅是示意性的。所属领域的技术人员可以清楚的了解到，为了描述的方便和简洁，在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。在本申请实施例、权利要求以及附图中揭示的特征可以独立存在也可以组合存在。在本申请实施例中以硬件形式描述的特征可以通过软件来执行，反之亦然。在此不做限定。
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应以所述权利要求的保护范围为准。

Claims (42)

  1. 一种样本数据标注系统，其特征在于，包括边缘节点和中心节点；
    所述边缘节点，用于：
    获取样本数据的关键特征；
    根据所述关键特征判断所述样本数据是否为未知样本数据；
    当所述样本数据为未知样本数据时，对所述未知样本数据进行标注处理，得到第一标注结果；
    将所述第一标注结果发送至所述中心节点；
    所述中心节点，用于：
    接收所述边缘节点发送的所述第一标注结果；
    当所述第一标注结果指示对所述未知样本数据标注成功时，对所述第一标注结果进行一致性处理，得到第二标注结果
  2. 根据权利要求1所述的系统，其特征在于，当所述第一标注结果指示对所述未知样本数据标注失败时，对所述未知样本数据进行标注处理，得到第三标注结果。
  3. 根据权利要求1或2所述的系统，其特征在于，当根据所述关键特征判断所述样本数据是否为未知样本数据时，所述边缘节点用于：
    根据未知样本模型和所述关键特征判断所述样本数据是否为未知样本数据。
  4. 根据权利要求3所述的系统，其特征在于，所述未知样本模型是根据从已知标注结果集中获取的多个关键特征生成的，所述已知标注结果集包括所述中心节点在生成所述未知样本模型前获取到的标注结果。
  5. 根据权利要求3或4所述的系统，其特征在于，所述未知样本模型包括多维坐标空间，根据所述样本数据的关键特征或所述关键特征的映射特征，得到坐标距离，根据所述坐标距离表示的相关性，确定所述样本数据是否为未知样本数据。
  6. 根据权利要求3或4所述的系统，其特征在于，所述未知样本模型包括神经网络模型。
  7. 根据权利要求1-6中任意一项所述的系统，其特征在于，
    当所述第一标注结果指示对所述未知样本数据标注成功时，所述第一标注结果包括：所述样本数据的标识、所述样本数据的样本特征和所述边缘节点为所述样本数据确定的标签；
    当所述第一标注结果指示对所述未知样本数据标注失败时，所述第一标注结果包括：所述样本数据的标识和所述样本数据的样本特征。
  8. 根据权利要求7所述的系统，其特征在于，所述第一标注结果的样本特征包括 所述样本数据和/或所述关键特征。
  9. 根据权利要求1-6中任意一项所述的系统，其特征在于，当对所述第一标注结果进行一致性处理时，所述中心节点用于通过相似度划分，将包括所述第一标注结果的多个第一标注结果进行聚类，得到所述第一标注结果对应的组。
  10. 根据权利要求9所述的系统，其特征在于，所述中心节点通过非监督的方式将包括所述第一标注结果的多个第一标注结果进行聚类，所述非监督方式包括如下一种或多种：K-MEANS、KNN。
  11. 根据权利要求9或10所述的系统，其特征在于，所述中心节点还用于，当所述组中的各标注结果的标签不一致时，对所述组中的所有标注结果的标签执行集成决策得到组标签，根据所述组标签得到所述第二标注结果。
  12. 根据权利要求11所述的系统，其特征在于，所述集成决策包括：投票法或加权投票法。
  13. 根据权利要求2或4所述的系统，其特征在于，所述中心节点，还用于根据所述第二标注结果和/或所述第三标注结果生成新的未知样本模型或更新所述未知样本数据模型。
  14. 一种样本数据标注方法，其特征在于，所述方法用于样本数据标注系统的边缘节点，所述方法包括：
    获取样本数据的关键特征；
    根据所述关键特征判断所述样本数据是否为未知样本数据；
    当所述样本数据为未知样本数据时，对所述未知样本数据进行标注处理，得到第一标注结果；
    将所述第一标注结果发送至所述样本数据标注系统的中心节点。
  15. 根据权利要求14所述的方法，其特征在于，所述据所述关键特征判断所述样本数据是否为未知样本数据包括：
    根据未知样本模型和所述关键特征判断所述样本数据是否为未知样本数据。
  16. 根据权利要求15所述的方法，其特征在于，所述未知样本模型是根据从已知标注结果集中获取的多个关键特征生成的，所述已知标注结果集包括所述中心节点在生成所述未知样本模型前获取到的标注结果。
  17. 根据权利要求15所述的方法，其特征在于，所述未知样本模型包括多维坐标空间，根据所述样本数据的关键特征或所述关键特征的映射特征，得到坐标距离，根 据所述坐标距离表示的相关性，确定所述样本数据是否为未知样本数据。
  18. 根据权利要求15所述的方法，其特征在于，所述未知样本模型包括神经网络模型。
  19. 根据权利要求14-18中任意一项所述的方法，其特征在于，当所述第一标注结果指示对所述未知样本数据标注成功时，所述第一标注结果包括：所述样本数据的样本标识、所述样本数据的样本特征和所述边缘节点为所述样本数据确定的标签；
    当所述第一标注结果指示对所述未知样本数据标注失败时，所述第一标注结果包括：所述样本数据的样本标识和所述样本数据的样本特征。
  20. 根据权利要求19所述的方法，其特征在于，所述第一标注结果的样本特征包括所述样本数据和/或所述关键特征。
  21. 一种样本数据标注方法，其特征在于，所述方法用于样本标注系统的中心节点，所述方法包括：
    接收所述样本标注系统中的边缘节点发送的第一标注结果，所述第一标注结果是由所述边缘节点对未知样本数据进行标注处理得到的；
    当所述第一标注结果指示对所述未知样本数据标注成功时，对所述第一标注结果进行一致性处理，得到第二标注结果。
  22. 根据权利要求21所述的方法，其特征在于，当所述标注结果指示对所述未知样本数据标注失败时，对所述未知样本数据进行标注处理，得到第三标注结果。
  23. 根据权利要求21或22所述的方法，其特征在于，当对所述第一标注结果进行一致性处理时，通过相似度划分，将包括所述第一标注结果的多个第一注册结果进行聚类，得到所述第一标注结果对应的组。
  24. 根据权利要求23所述的方法，其特征在于，通过非监督的方式，将包括所述第一标注结果的多个第一标注结果进行聚类，所述非监督方式包括如下一种或多种：K-MEANS、KNN。
  25. 根据权利要求23或24所述的方法，其特征在于，当所述组中的各标注结果的标签不一致时，对所述组中的所有标注结果的标签执行集成决策得到组标签，根据所述组标签得到所述第二标注结果。
  26. 根据权利要求25所述的方法，其特征在于，所述集成决策包括：投票法或加权投票法。
  27. 根据权利要求21-26中任意一项所述的方法，其特征在于，所述方法还包括：根据所述第二标注结果和/或所述第三标注结果生成新的未知样本模型或更新所述中心节点已有的未知样本数据模型。
  28. 一种节点，其特征在于，所述节点部署在样本数据标注系统中，所述节点包括：
    获取单元，用于获取样本数据的关键特征；
    识别单元，用于识别未知样本数据，根据所述关键特征判断所述样本数据是否为未知样本数据；
    标注单元，当所述样本数据为未知样本数据时，对所述未知样本数据进行标注处理，得到第一标注结果。
  29. 根据权利要求28所述的节点，其特征在于，还包括：
    通信单元，用于将所述第一标注结果发送至所述样本数据标注系统的中心节点。
  30. 根据权利要求28或29所述的节点，其特征在于，所述识别单元还用于：根据未知样本模型和所述关键特征判断所述样本数据是否为未知样本数据。
  31. 根据权利要求30所述的节点，其特征在于，所述未知样本模型是根据从已知标注结果集中获取的多个关键特征生成的，所述已知标注结果集包括所述中心节点在生成所述未知样本模型前获取到的标注结果。
  32. 根据权利要求31所述的节点，其特征在于，所述未知样本模型包括多维坐标空间，根据所述样本数据的关键特征或所述关键特征的映射特征，得到坐标距离，根据所述坐标距离表示的相关性，确定所述样本数据是否为未知样本数据。
  33. 根据权利要求31所述的节点，其特征在于，所述未知样本模型包括神经网络模型。
  34. 根据权利要求28-33中任意一项所述的节点，其特征在于，当所述第一标注结果指示对所述未知样本数据标注成功时，所述第一标注结果包括：所述样本数据的样本标识、所述样本数据的样本特征和所述边缘节点为所述样本数据确定的标签；
    当所述第一标注结果指示对所述未知样本数据标注失败时，所述第一标注结果包括：所述样本数据的样本标识和所述样本数据的样本特征。
  35. 根据权利要求34所述的节点，其特征在于，所述第一标注结果的样本特征包括所述样本数据和/或所述关键特征。
  36. 一种节点，其特征在于，所述节点部署在样本数据标注系统中，所述节点包 括：
    通信单元，所述通信单元用于接收所述样本标注系统中的边缘节点发送的第一标注结果，所述第一标注结果是由所述边缘节点对未知样本数据进行标注处理得到的；
    处理单元，当所述第一标注结果指示对所述未知样本数据标注成功时，所述处理单元对所述第一标注结果进行一致性处理，得到第二标注结果。
  37. 根据权利要求36所述的节点，其特征在于，所述节点还包括标注单元，当所述标注结果指示对所述未知样本数据标注失败时，所述标注单元对所述未知样本数据进行标注处理，得到第三标注结果。
  38. 根据权利要求36或37所述的节点，其特征在于，当对所述第一标注结果进行一致性处理时，所述处理单元通过相似度划分，将包括所述第一标注结果的多个第一标注结果进行聚类，得到所述第一标注结果对应的组。
  39. 根据权利要求38所述的节点，其特征在于，所述处理单元通过非监督的方式，将包括所述第一标注结果的多个第一标注结果进行聚类，所述非监督方式包括如下一种或多种：K-MEANS、KNN。
  40. 根据权利要求38或39所述的节点，其特征在于，当所述组中的各标注结果的标签不一致时，所述处理单元对所述组中的所有标注结果的标签执行集成决策得到组标签，根据所述组标签得到所述第二标注结果。
  41. 根据权利要求40所述的节点，其特征在于，所述集成决策包括：投票法或加权投票法。
  42. 根据权利要求36-41所述的节点，其特征在于，所述节点还包括模型单元，所述模型单元根据所述第二标注结果和/或所述第三标注结果生成新的未知样本模型或更新所述中心节点已有的未知样本数据模型。
PCT/CN2021/095786 2020-07-06 2021-05-25 样本数据标注系统、方法以及相关设备 Ceased WO2022007527A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP21837643.2A EP4163803A4 (en) 2020-07-06 2021-05-25 SAMPLE DATA ANNOTATION SYSTEM, METHOD AND APPARATUS
US18/150,505 US20230169096A1 (en) 2020-07-06 2023-01-05 Sample Data Annotation System and Method, and Related Device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010642572.5A CN113971422B (zh) 2020-07-06 2020-07-06 样本数据标注系统、方法以及相关设备
CN202010642572.5 2020-07-06

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/150,505 Continuation US20230169096A1 (en) 2020-07-06 2023-01-05 Sample Data Annotation System and Method, and Related Device

Publications (1)

Publication Number Publication Date
WO2022007527A1 true WO2022007527A1 (zh) 2022-01-13

Family

ID=79553669

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/095786 Ceased WO2022007527A1 (zh) 2020-07-06 2021-05-25 样本数据标注系统、方法以及相关设备

Country Status (4)

Country Link
US (1) US20230169096A1 (zh)
EP (1) EP4163803A4 (zh)
CN (1) CN113971422B (zh)
WO (1) WO2022007527A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114462020A (zh) * 2022-04-11 2022-05-10 广州卓远虚拟现实科技有限公司 基于区块链的软件授权方法及软件授权系统
CN115469996A (zh) * 2022-07-28 2022-12-13 天翼云科技有限公司 算力调度方法、装置、电子设备及存储介质

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20230222182A1 (en) * 2022-01-11 2023-07-13 Dell Products L.P. Unknown object classification for unsupervised scalable auto labelling
CN118093810A (zh) * 2024-01-23 2024-05-28 浙江阿里巴巴机器人有限公司 数据处理方法、数据处理模型训练方法及样本集构建方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030005152A1 (en) * 2001-03-09 2003-01-02 Arif Diwan Content-request redirection method and system
US20150356182A1 (en) * 2014-06-04 2015-12-10 Tektronix, Inc. Bandwidth efficient processing and filtering across distributed databases
CN109710933A (zh) * 2018-12-25 2019-05-03 广州天鹏计算机科技有限公司 训练语料的获取方法、装置、计算机设备和存储介质
CN110442519A (zh) * 2019-07-23 2019-11-12 深圳市六度人和科技有限公司 一种崩溃文件处理方法、装置、电子设备及存储介质
CN110730245A (zh) * 2019-10-22 2020-01-24 青岛农业大学 基于神经网络的边缘计算系统和方法
CN111125124A (zh) * 2019-11-18 2020-05-08 云知声智能科技股份有限公司 一种基于大数据平台的语料标注的方法及装置

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106355628B (zh) * 2015-07-16 2019-07-05 中国石油化工股份有限公司 图文知识点标注方法和装置、图文标注的修正方法和系统
AU2018269941A1 (en) * 2017-05-14 2019-12-05 Digital Reasoning Systems, Inc. Systems and methods for rapidly building, managing, and sharing machine learning models
NZ759818A (en) * 2017-10-16 2022-04-29 Illumina Inc Semi-supervised learning for training an ensemble of deep convolutional neural networks
CN109446783B (zh) * 2018-11-16 2023-07-25 山东浪潮科学研究院有限公司 一种基于机器众包的图像识别高效样本采集方法及系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030005152A1 (en) * 2001-03-09 2003-01-02 Arif Diwan Content-request redirection method and system
US20150356182A1 (en) * 2014-06-04 2015-12-10 Tektronix, Inc. Bandwidth efficient processing and filtering across distributed databases
CN109710933A (zh) * 2018-12-25 2019-05-03 广州天鹏计算机科技有限公司 训练语料的获取方法、装置、计算机设备和存储介质
CN110442519A (zh) * 2019-07-23 2019-11-12 深圳市六度人和科技有限公司 一种崩溃文件处理方法、装置、电子设备及存储介质
CN110730245A (zh) * 2019-10-22 2020-01-24 青岛农业大学 基于神经网络的边缘计算系统和方法
CN111125124A (zh) * 2019-11-18 2020-05-08 云知声智能科技股份有限公司 一种基于大数据平台的语料标注的方法及装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4163803A4

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114462020A (zh) * 2022-04-11 2022-05-10 广州卓远虚拟现实科技有限公司 基于区块链的软件授权方法及软件授权系统
CN114462020B (zh) * 2022-04-11 2022-07-12 广州卓远虚拟现实科技有限公司 基于区块链的软件授权方法及软件授权系统
CN115469996A (zh) * 2022-07-28 2022-12-13 天翼云科技有限公司 算力调度方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
EP4163803A4 (en) 2023-12-13
US20230169096A1 (en) 2023-06-01
CN113971422A (zh) 2022-01-25
CN113971422B (zh) 2026-02-06
EP4163803A1 (en) 2023-04-12

Similar Documents

Publication Publication Date Title
CN111062495B (zh) 机器学习方法及相关装置
CN113971422B (zh) 样本数据标注系统、方法以及相关设备
CN112199462A (zh) 跨模态的数据处理方法、装置、存储介质以及电子装置
CN112602304A (zh) 基于行为属性标识设备类型
CN107077487A (zh) 利用深度网络对个人照片加标签
CN108108499A (zh) 人脸检索方法、装置、存储介质及设备
CN113821667B (zh) 基于人工智能的图像处理方法、装置及电子设备
CN111368926A (zh) 图像筛选方法、装置和计算机可读存储介质
CN111709473A (zh) 对象特征的聚类方法及装置
CN115244545A (zh) 用于多模式分布式学习的模型池
US20210176181A1 (en) Intelligent Conversion of Internet Domain Names to Vector Embeddings
CN107240029A (zh) 一种数据处理方法及装置
CN116032741A (zh) 一种设备识别方法、装置、电子设备和计算机存储介质
CN115168609A (zh) 一种文本匹配方法、装置、计算机设备和存储介质
CN113315851A (zh) 域名检测方法、装置及存储介质
CN110390356B (zh) 视觉词典生成方法及装置、存储介质
CN113822295B (zh) 图像识别方法、装置、电子设备和计算机可读存储介质
CN114358202A (zh) 基于药物分子图像分类的信息推送方法及装置
CN110135428A (zh) 图像分割处理方法和装置
CN112115957A (zh) 数据流识别方法及装置、计算机存储介质
WO2024169529A1 (zh) 知识库的构建方法、数据检索方法、装置和云设备
CN110276283B (zh) 图片识别方法、目标识别模型训练方法及装置
CN109361529A (zh) 一种终端类型识别方法、装置、电子设备及存储介质
CN114495175B (zh) 一种行人重识别模型创建方法、装置、设备及存储介质
CN114912572A (zh) 目标识别方法及神经网络的训练方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21837643

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021837643

Country of ref document: EP

Effective date: 20230109

NENP Non-entry into the national phase

Ref country code: DE