WO2020020098A1 - 网络流测量的方法、网络测量设备以及控制面设备 - Google Patents

网络流测量的方法、网络测量设备以及控制面设备 Download PDF

Info

Publication number
WO2020020098A1
WO2020020098A1 PCT/CN2019/097092 CN2019097092W WO2020020098A1 WO 2020020098 A1 WO2020020098 A1 WO 2020020098A1 CN 2019097092 W CN2019097092 W CN 2019097092W WO 2020020098 A1 WO2020020098 A1 WO 2020020098A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
stream
measurement information
type
measurement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2019/097092
Other languages
English (en)
French (fr)
Inventor
黄勇
黄群
张弓
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to EP19840256.2A priority Critical patent/EP3823217A4/en
Publication of WO2020020098A1 publication Critical patent/WO2020020098A1/zh
Priority to US17/158,586 priority patent/US11706114B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/02—Capturing of monitoring data
    • H04L43/026—Capturing of monitoring data using flow identification
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0876—Network utilisation, e.g. volume of load or congestion level
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/142—Network analysis or design using statistical or mathematical methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L63/00—Network architectures or network communication protocols for network security
    • H04L63/14—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic
    • H04L63/1408—Network architectures or network communication protocols for network security for detecting or protecting against malicious traffic by monitoring network traffic

Definitions

  • the present application relates to the field of information technology, and more particularly, to a method for network flow measurement, a network measurement device, and a control plane device.
  • the basis for network intelligence is the entire network's full network data measurement.
  • two key requirements need to be met. First, high performance, that is, information extraction and statistics based on each message; second, universality, that is, when there are multiple aspects of measurement data required for network flow, a small number of general-purpose sensors can be used to support many Measurement target.
  • the present application provides a network flow measurement method, a network measurement device, and a control plane device, which can perform measurement based on the bit level of a flow key and record measurement information, thereby realizing the universality of network flow measurement.
  • an embodiment of the present application provides a network flow measurement method, which is characterized in that the method is applicable to a system including a network measurement device and a control plane device, and the method includes: the network measurement device measures First data, the first data including a first type of data structure, the first type of data structure including first measurement information, the first measurement information corresponding to a bit of a key of the stream; the network measurement The device sends the first data to the control plane device.
  • the network measurement device performs flow data measurement by using a bit-level first-level data structure corresponding to a keyword of a flow included in the first data, and reports measurement information based on the first data record to Control surface equipment. Therefore, a universal measurement method can be used to support the measurement of multiple target flows such as large flows, arbitrary flows, and sudden flows, and it is not affected by the flow distribution. A single sensor is used to support a wide range of network flow measurements. application.
  • the first type of data structure includes measurement information of at least one bit of a key of a corresponding stream.
  • the key corresponding to the stream is 4 bits
  • the first type of data structure may include 3 or 2 bits of measurement information corresponding to the 4 bits of the stream key, that is, the first type
  • the data structure may include measurement information of a portion of bits corresponding to a key.
  • the first data further includes a second type of data structure, and the second type of data structure includes second measurement information of the stream.
  • the first data may further include a second type of data structure, and keywords of all streams may be recorded in the second type of data structure.
  • the first type of data structure and the second type of data structure constitute a network measurement.
  • the first type of data structure includes measurement information corresponding to each bit of the keyword.
  • the key corresponding to the stream is 4 bits
  • the first type of data structure may include measurement information corresponding to 4 bits of the key of the stream, that is, the first type of data structure may include the key corresponding to the key. Measurement information of the 4 bits of the word.
  • the first type of data structure is a matrix of K first data units, where each first data unit matrix is used according to a key of the stream A value of one bit records the first measurement information;
  • the second type of data structure is a second data unit matrix, the second data unit matrix is used to record the second measurement information, and K is greater than Or a positive integer equal to 1.
  • each of the K first data unit matrices corresponds to one bit of a key of a stream.
  • the four first data unit matrices correspond to the key of the stream, that is, the first bit of the key corresponds to the first first data unit matrix, and the second bit of the key Corresponds to the second first data unit matrix, the third bit corresponds to the third first data unit matrix, and the fourth bit corresponds to the fourth first data unit matrix.
  • the data unit matrix may be a data structure including r rows and c columns, where r * c data units are included.
  • the number of first matrix elements included in the first type of data structure is related to the number of bits of the key of the corresponding stream. If the key of the corresponding stream includes K bits, the first type of data
  • the structure may include 1 to K first data unit matrices, where the first first data unit matrix records measurement information corresponding to a first bit of a key of a corresponding stream, and the Kth first data unit matrix corresponds to The first bit of the key of the stream records the measurement information.
  • the second type of data unit structure includes a second data unit matrix. The second data unit matrix records the measurement information based on the keywords of the corresponding stream. Keyword's overall record measurement information.
  • the measurement information may be the number of packets, or the length of the packets, or parameters related to the attributes of the packets.
  • the method further includes:
  • the network measurement device records the second measurement information of the first stream in M ij in the second data unit matrix, where M ij represents a data unit in the i-th row and the j-th column, and i is less than or equal to a positive integer of r, j being a positive integer less than or equal to c, and the second measurement information including at least one of the number of packets or the length of the packets in the first stream;
  • the measurement device When the value of the Nth bit of the first stream keyword is a preset value, the measurement device records the first stream ’s M ij in the Nth data unit matrix of the first type of data structure. Said first measurement information, said first measurement information including at least one of a number of messages or a message length in said first stream, said N being less than or equal to said K, said K and said N both being Is a positive integer.
  • the preset value is 1.
  • the method further includes: M ij of the network measurement device in the second data unit matrix records a first-line statistical value, where M ij Represents a data unit in the i-th row and j-th column, where i is a positive integer less than or equal to r, j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message.
  • the measurement device records the Mth ij of the Nth data unit matrix in the first type of data structure A first-class statistical value, the N is less than or equal to the K, and the K and the N are both positive integers.
  • M ij in the second data unit matrix of the network measurement device records the first-class statistical value, where M ij represents a data unit in the i-th row and the j-th column, where i is A positive integer less than or equal to r, the j being a positive integer less than or equal to c, and the statistical value including at least one of the number of messages or the length of the message;
  • M ij of the Nth data unit matrix in the first type of data structure of the measurement device records the statistical value of the first stream, where N is less than or equal to K, and K and the N are both positive integers.
  • M ij in the second data unit matrix of the network measurement device records the first-class statistical value, where M ij represents a data unit in the i-th row and the j-th column, where i is A positive integer less than or equal to r, the j being a positive integer less than or equal to c, and the statistical value including at least one of the number of messages or the length of the message;
  • M ij of the Nth data unit matrix in the first type data structure of the measurement device records the statistical value of the first stream, where N is less than or equal to K, and K and the N are both positive integers.
  • the position of the data unit that records the statistics of the first stream based on the bits of the keywords of the corresponding stream in the first type of data structure and the keywords of the corresponding stream based on the keywords of the corresponding stream in the second type of data structure records the statistics of the first stream as a whole
  • the location of the data unit is the same.
  • the second type of data structure is the second data unit matrix. If the keywords of the first stream record the statistical values in the i-th row and the j-th column in the second data unit matrix, then the K first data unit matrix The i-th row and the j-th column record the statistics of the bits corresponding to the first-rate keywords.
  • M ij can record the statistics of multiple streams. These multiple streams have a common feature, that is, they are indexed to the same data unit in the data unit matrix according to the hash value by the same hash function.
  • an embodiment of the present application provides a network flow measurement method.
  • the method is applicable to a system including a network measurement device and a control plane device.
  • the method includes: the control plane device acquires the network measurement First data reported by a device, the first data including a first type number, the first type data structure including first measurement information, the first measurement information corresponding to a bit of a keyword of the stream; the The control plane device obtains a statistical result according to the first data.
  • the control plane device receives the measurement information reported by the network measurement device, and the network measurement device measures the stream data by using the first type data structure of the bit level corresponding to the key of the stream included in the first data, and The measurement information based on the first data record is reported to the control plane device, and the control plane device obtains a statistical result according to the probability distribution characteristic of the first data. Therefore, a universal measurement method can be used to support the measurement of multiple target flows such as large flows, arbitrary flows, and sudden flows, and it is not affected by the flow distribution. A single sensor is used to support a wide range of network flow measurements. application.
  • the first type of data structure includes measurement information of at least one bit of a key of a corresponding stream.
  • the key corresponding to the stream is 4 bits
  • the first type of data structure may include 3 or 2 bits of measurement information corresponding to the 4 bits of the stream key, that is, the first type
  • the data structure may include measurement information of a portion of bits corresponding to a key.
  • the first data further includes a second type of data structure, and the second type of data structure includes measurement information corresponding to keywords of the stream.
  • the first data further includes a second type of data structure, and the second type of data structure includes second measurement information of the stream, and the second The measurement information corresponds to a key of the stream.
  • the first data may further include a second type of data structure, and keywords of all streams may be recorded in the second type of data structure.
  • the first type of data structure and the second type of data structure constitute a network measurement.
  • the first measurement information corresponds to each bit of a key of the stream.
  • the first type of data structure includes measurement information corresponding to each bit of the keyword.
  • the key corresponding to the stream is 4 bits
  • the first type of data structure may include measurement information corresponding to 4 bits of the key of the stream, that is, the first type of data structure may include the key corresponding to the key Measurement information of the 4 bits of the word.
  • the first type of data structure is a matrix of K first data units, where each first data unit matrix is used according to a key of the stream The value of the bit records the first measurement information;
  • the second type of data structure is a second data unit matrix, the second data unit matrix is used to record the second measurement information, and K is greater than or A positive integer equal to 1.
  • the number of first matrix elements included in the first type of data structure is related to the number of bits of the key of the corresponding stream. If the key of the corresponding stream includes K bits, the first type of data
  • the structure may include 1 to K first data unit matrices, where the first first data unit matrix records measurement information corresponding to a first bit of a key of a corresponding stream, and the Kth first data unit matrix corresponds to The first bit of the key of the stream records the measurement information.
  • the second type of data unit structure includes a second data unit matrix. The second data unit matrix records the measurement information based on the keywords of the corresponding stream. Keyword's overall record measurement information.
  • the measurement information may be the number of packets, or the length of the packets, or parameters related to the attributes of the packets.
  • M ij in the second type of data structure is used to record the second measurement information of the first stream, where M ij represents an i-th row and a j-th row Data unit of the column, where i is a positive integer less than or equal to r, j is a positive integer less than or equal to c, and the second measurement information includes the number of messages in the first stream or the message length At least one of
  • the first type of data structure is K first data unit matrices, where M ij in the Nth data unit matrix is used when the value of the Nth bit of the first stream keyword is a preset value, Record the first measurement information of the first stream, the second measurement information includes at least one of the number of messages or the message length in the first stream, and the N is less than or equal to the K, so Let N be a positive integer.
  • the first measurement information and the second measurement information may be statistical values.
  • M ij in the second type of data structure is used to record the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column , I is a positive integer less than or equal to r, j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message;
  • the first type of data structure Are K first data unit matrices, where M ij in the Nth data unit matrix is used to record statistics of the first stream when the value of the Nth bit of the first stream keyword is a preset value Value, the N is less than or equal to the K, and the K and the N are both positive integers.
  • M ij in the second type of data structure is used to record the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column, where i is less than or equal to r A positive integer of, j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message;
  • the first type of data structure is a matrix of K first data units, Wherein, M ij in the Nth data unit matrix is used to record the statistical value of the first stream when the value of the Nth bit of the first stream keyword is 1, the N is less than or equal to the K , K and N are both positive integers.
  • M ij in the second type of data structure is used to record the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column, where i is less than or equal to r A positive integer of, j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message;
  • the first type of data structure is a matrix of K first data units, Wherein, M ij in the Nth data unit matrix is used to record the statistical value of the first stream when the value of the Nth bit of the first stream keyword is 0, where N is less than or equal to the K , K and N are both positive integers.
  • M ij can record the statistics of multiple streams. These multiple streams have a common feature, that is, they are indexed to the same data unit in the data unit matrix according to the hash value by the same hash function.
  • the control plane device obtains a statistical result according to the first data, including: the control plane device according to a distribution on each bit of the keyword A characteristic to obtain a first set and the updated first data, the first set including a large stream extracted from r * c stacks included in the first data, and the updated first data Including all data units corresponding to the r * c stack stack except the residual stream of the first set, any one of the r * c stacks is the same according to the K + 1 data unit matrix A data unit group consisting of rows and columns of data units.
  • the streams with the same hash conflict can be recorded together in the same rows and columns in the K + 1 data unit matrix.
  • the K + 1 data units can be a stack
  • (i, j) indicates that a data unit group consisting of data units in the i-th row and j column in the K + 1 data unit matrix is a stack stack, so that the first data may include r * c stack stacks.
  • the acquiring, by the control plane device, a first set and updated first data according to a distribution characteristic on each bit of the keyword includes: The control plane device extracts a large stream from the first stack of the r * c stack stacks, the first set includes the large stream extracted from the first stack, and the updated first data Including all data units corresponding to the r * c stack stacks except the residual stream extracted from the first stack.
  • the method further includes: if the updated first data satisfies a normal distribution, determining that the first set includes the first data All big streams in.
  • the method further includes: if the updated first data does not satisfy a normal distribution, then the r included in the first data * c A large stream is extracted from the stack.
  • the stack (i, j) does not include a large flow
  • its probability distribution follows a normal distribution.
  • the probability distribution in the updated first data does not satisfy the normal distribution, it means that the updated first data also includes a large stream, so it is necessary to continue to extract the large stream from the updated first data.
  • the method further includes: if the first set is an empty set, adjusting the threshold from the r * c numbers included in the first data A large flow is extracted in a stack, wherein the threshold value represents a ratio of a large flow flow to a total flow or a ratio of a large flow frequency to a total flow frequency.
  • the threshold may represent the minimum ratio of the large flow rate to the total flow rate, or may represent the minimum ratio of the large flow rate to the total flow rate.
  • extracting a large stream from the r * c stack stacks included in the first data after adjusting the threshold includes: obtaining the r * The probability that the value of one bit in the c stacks is 1, the third set is determined according to the probability, and the third set includes candidate keywords of the large flow; the large flow is determined according to the third set Keywords.
  • the determining the keyword of the large stream according to the third set includes: based on each bit of the first type data structure and the first The ratio of the same stack in the second type of data structure determines the K frequencies, and the frequency of at least one keyword included in the third set is calculated; the at least one keyword is filtered according to the certainty probability of each bit to determine all The keywords of the large stream are described; frequency checking is performed on the keywords of the large stream according to a hash function.
  • a candidate keyword set may be determined according to a probability that a value of a corresponding bit in the stack is 1. For example, the ratio of the frequency value recorded in the k-th first data unit matrix to the frequency data in the second data unit matrix in the stack (i, j) before extracting the large stream is recorded as R i, j [k ] . If R i, j [k] > ⁇ , and 1-R i, j [k] > ⁇ (for example, ⁇ is 0.99), it is estimated that the k- th bit is 1 or 0. If not, we mark the k-th bit as *, which means it can be either 1 or 0.
  • the statistical result includes a list of large flows in the first data, or data of residual flows other than large flows in the first data, or At least one of the distributions of frequency of stream data per bit in the first data.
  • the method further includes: the control plane device receives a traffic query request; the control plane device sends a query result according to the traffic query request.
  • the query request obtained by the control plane device may be a query request originating from a third-party application.
  • an embodiment of the present application provides a network measurement device, which is applicable to a system including the network measurement device and a control plane device, and the network measurement device includes a processing module for measuring First data, the first data including a first type of data structure, the first type of data structure including first measurement information of a stream, the first measurement information corresponding to a bit of a key of the stream; a communication module Is configured to send the first data to the control plane device.
  • the network measurement device performs flow data measurement by using a bit-level first-level data structure corresponding to a keyword of a flow included in the first data, and reports measurement information based on the first data record to Control surface equipment. Therefore, a universal measurement method can be used to support the measurement of multiple target flows such as large flows, arbitrary flows, and sudden flows, and it is not affected by the flow distribution. A single sensor is used to support a wide range of network flow measurements. application.
  • the first type of data structure includes measurement information of at least one bit of a key of a corresponding stream.
  • the first type of data structure may include 3 or 2 bits of measurement information corresponding to the 4 bits of the stream key, that is, the first The class data structure may include measurement information of a portion of bits corresponding to a key.
  • the first data further includes a second type of data structure, the second type of data structure includes second measurement information of the stream, and the second The measurement information corresponds to a key of the stream.
  • the first measurement information corresponds to each bit of a key of the stream.
  • the first type of data structure includes measurement information corresponding to each bit of the keyword.
  • the first type of data structure is a matrix of K first data units, where each first data unit matrix is used according to a key of the stream A value of one bit records the first measurement information;
  • the second type of data structure is a second data unit matrix, the second data unit matrix is used to record the second measurement information, and K is greater than Or a positive integer equal to 1.
  • the number of first matrix elements included in the first type of data structure is related to the number of bits of the key of the corresponding stream. If the key of the corresponding stream includes K bits, the first type of data
  • the structure may include 1 to K first data unit matrices, where the first first data unit matrix records measurement information corresponding to a first bit of a key of a corresponding stream, and the Kth first data unit matrix corresponds to The first bit of the key of the stream records the measurement information.
  • the second type of data unit structure includes a second data unit matrix. The second data unit matrix records the measurement information based on the keywords of the corresponding stream. Keyword's overall record measurement information.
  • the measurement information may be the number of packets, or the length of the packets, or parameters related to the attributes of the packets.
  • the processing module is specifically configured to record the first measurement information of the first stream in M ij in the second data unit matrix, where M ij represents a data unit in the i-th row and the j-th column, where i is a positive integer less than or equal to r, j is a positive integer less than or equal to c, and the first measurement information includes a report included in the first stream.
  • At least one of the number of messages or the length of the message when the value of the Nth bit of the first stream keyword is a preset value, M of the Nth data unit matrix in the first type of data structure ij records the second measurement information of the first stream, the N is less than or equal to the K, and the N is a positive integer.
  • the first measurement information and the second measurement information may be statistical values.
  • the preset value is 1.
  • M ij in the second data unit matrix of the network measurement device records the first-class statistical value, where M ij represents a data unit in the i-th row and the j-th column, where i is A positive integer less than or equal to r, the j being a positive integer less than or equal to c, and the statistical value including at least one of the number of messages or the length of the message;
  • M ij of the Nth data unit matrix in the first type of data structure of the measurement device records the statistical value of the first stream, where N is less than or equal to K, and K and the N are both positive integers.
  • the preset value is 0.
  • M ij in the second data unit matrix of the network measurement device records the first-class statistical value, where M ij represents a data unit in the i-th row and the j-th column, where i is A positive integer less than or equal to r, the j being a positive integer less than or equal to c, and the statistical value including at least one of the number of messages or the length of the message;
  • M ij of the Nth data unit matrix in the first type data structure of the measurement device records the statistical value of the first stream, where N is less than or equal to K, and K and the N are both positive integers.
  • an embodiment of the present application provides a control plane device.
  • the control plane device is applicable to a system including a network measurement device and the control plane device.
  • the control plane device includes a communication module for acquiring The first data reported by the network measurement device, the first data includes a first type number, the first type data structure includes first measurement information of a flow, and the first measurement information and a keyword of the flow Corresponding to the bit; the processing module obtains a statistical result according to the first data.
  • the control plane device receives the measurement information reported by the network measurement device, and the network measurement device measures the stream data by using the first type data structure of the bit level corresponding to the key of the stream included in the first data, and The measurement information based on the first data record is reported to the control plane device, and the control plane device obtains a statistical result according to the probability distribution characteristic of the first data. Therefore, a universal measurement method can be used to support the measurement of multiple target flows such as large flows, arbitrary flows, and sudden flows, and it is not affected by the flow distribution. A single sensor is used to support a wide range of network flow measurements. application.
  • the first type of data structure includes measurement information of at least one bit of a key of a corresponding stream.
  • the first type of data structure may include 3 or 2 bits of measurement information corresponding to the 4 bits of the stream key, that is, the first The class data structure may include measurement information of a portion of bits corresponding to a key.
  • the first data further includes a second type of data structure, and the second type of data structure includes second measurement information of the stream, and the second The measurement information corresponds to a key of the stream.
  • the first measurement information corresponds to each bit of a key of the stream.
  • the first type of data structure includes measurement information corresponding to each bit of the keyword.
  • the first type of data structure may include measurement information corresponding to the 5 bits of the key of the stream, that is, the first type of data structure may include the 5 bits corresponding to the key Measurement information for each bit.
  • the first type of data structure is a matrix of K first data units, where each first data unit matrix is used according to a key of the stream A value of one bit records the first measurement information;
  • the second type of data structure is a second data unit matrix, the second data unit matrix is used to record the second measurement information, and K is greater than Or a positive integer equal to 1.
  • the number of first matrix elements included in the first type of data structure is related to the number of bits of the key of the corresponding stream. If the key of the corresponding stream includes K bits, the first type of data
  • the structure may include 1 to K first data unit matrices, where the first first data unit matrix records measurement information corresponding to a first bit of a key of a corresponding stream, and the Kth first data unit matrix corresponds to The first bit of the key of the stream records the measurement information.
  • the second type of data unit structure includes a second data unit matrix. The second data unit matrix records the measurement information based on the keywords of the corresponding stream. Keyword's overall record measurement information.
  • the measurement information may be the number of packets, or the length of the packets, or parameters related to the attributes of the packets.
  • M ij in the second type of data structure is used to record the second measurement information of the first stream, where M ij represents an i-th row and a j-th row Data unit of the column, where i is a positive integer less than or equal to r, j is a positive integer less than or equal to c, and the second measurement information includes the number of messages in the first stream or the message length At least one of the above;
  • the first type of data structure is K first data unit matrices, where M ij in the Nth data unit matrix is used for the value of the Nth bit of the first stream key When the value is preset, the first measurement information of the first stream is recorded, and the first measurement information includes at least one of a number of messages or a message length in the first stream, and N is less than or equal to Let K be a positive integer.
  • the preset value is 1.
  • M ij in the second type of data structure is used to record the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column, where i is less than or equal to r A positive integer of, j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message;
  • the first type of data structure is a matrix of K first data units, Wherein, M ij in the Nth data unit matrix is used to record the statistical value of the first stream when the value of the Nth bit of the first stream keyword is 1, the N is less than or equal to the K , K and N are both positive integers.
  • the preset value is 0.
  • M ij in the second type of data structure is used to record the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column, where i is less than or equal to r A positive integer of, j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message;
  • the first type of data structure is a matrix of K first data units, Wherein, M ij in the Nth data unit matrix is used to record the statistical value of the first stream when the value of the Nth bit of the first stream keyword is 0, where N is less than or equal to the K , K and N are both positive integers.
  • M ij can record the statistics of multiple streams. These multiple streams have a common feature, that is, they are indexed to the same data unit in the data unit matrix according to the hash value by the same hash function.
  • the processing module is specifically configured to obtain a first set and the updated first data according to a distribution characteristic on each bit of the keyword, so
  • the first set includes a large stream extracted from the r * c stack stacks included in the first data
  • the updated first data includes the r * c stack stacks except the first All data units corresponding to the set of residual streams
  • any one of the r * c stacks is a data unit group composed of data units of the same row and column in a K + 1 data unit matrix.
  • the streams with the same hash conflict can be recorded together in the same rows and columns in the K + 1 data unit matrix.
  • the K + 1 data units can be a stack
  • (i, j) indicates that a data unit group consisting of data units in the i-th row and j column in the K + 1 data unit matrix is a stack stack, so that the first data may include r * c stack stacks.
  • the processing module is specifically configured to extract a large stream from a first stack of the r * c stack stacks, and the first set includes The large stream extracted in the first stack, and the updated first data includes all data corresponding to the residual stream in the r * c stack stacks except for the residual stream extracted in the first stack. unit.
  • the processing module is further configured to determine that the first set includes the first All big streams in the data.
  • the stack (i, j) does not include a large flow
  • its probability distribution follows a normal distribution.
  • the probability distribution in the updated first data does not satisfy the normal distribution, it means that the updated first data also includes a large stream, so it is necessary to continue to extract the large stream from the updated first data.
  • the processing module is further configured to obtain the updated data from the first data.
  • a large stream is extracted from r * c stacks.
  • the processing module is specifically configured to adjust the threshold from the r * c included in the first data A large flow is extracted from each stack, wherein the threshold value represents a ratio of a large flow flow to a total flow or a ratio of a large flow frequency to a total flow frequency.
  • the threshold may represent the minimum ratio of the large flow rate to the total flow rate, or may represent the minimum ratio of the large flow rate to the total flow rate.
  • the processing module is specifically configured to obtain a probability that the value of each bit in the r * c stacks is 1; and determine the first Three sets, the third set includes candidate keywords of the big stream; and the keywords of the big stream are determined according to the third set.
  • the processing module is specifically configured to determine K based on a ratio of the first type of data structure to the same stack in the second type of data structure per bit Frequency, calculating the frequency of at least one keyword included in the third set; filtering the at least one keyword according to each bit certainty probability to determine the keyword of the high flow; according to a hash function Perform a frequency check on the keywords of the large flow.
  • a candidate keyword set may be determined according to a probability that a value corresponding to each bit in the stack is 1. For example, the ratio of the frequency value recorded in the k-th first data unit matrix to the frequency data in the second data unit matrix in the stack (i, j) before extracting the large stream is recorded as R i, j [k ] . If R i, j [k] > ⁇ , and 1-R i, j [k] > ⁇ (for example, ⁇ is 0.99), it is estimated that the k- th bit is 1 or 0. If not, we mark the k-th bit as *, which means it can be either 1 or 0.
  • the statistical result includes a list of large flows in the first data, or data of residual flows other than large flows in the first data, or At least one of the distributions of frequency of stream data per bit in the first data.
  • the communication module is further configured to receive a traffic query request by the control plane device; and the processing module is further configured to send a query according to the traffic query request result.
  • the query request obtained by the control plane device may be a query request originating from a third-party application.
  • a network measurement device includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the apparatus executes the first Aspect or any one of the possible ways to implement the method in the first aspect.
  • an embodiment of the present application provides a control plane device.
  • the control plane device includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the apparatus executes
  • the second aspect or the method in any one of the second aspects may be implemented.
  • a system includes at least one network measurement device in any one of the foregoing aspects or any possible implementation manner of any one of the aspects and any one of the foregoing aspects or any one of the aspects. Possible implementation of a control plane device.
  • an embodiment of the present application provides a computer-readable storage medium that stores instructions for implementing the first aspect or the method described in any possible implementation manner of the first aspect.
  • an embodiment of the present application provides a computer-readable storage medium that stores instructions for implementing the second aspect or the method described in any possible implementation manner of the second aspect.
  • the present application provides a computer program product containing instructions, and when the computer program product runs on a computer, the computer is caused to execute the first aspect or any possible implementation manner of the first aspect. method.
  • the present application provides a computer program product containing instructions, and when the computer program product runs on a computer, the computer is caused to execute the second aspect or any possible implementation manner of the second aspect.
  • Figure 1 shows a schematic diagram of the data structure of a sketch-based measurement technique.
  • FIG. 2 is a schematic diagram of a method for recording a stream in a network device in the prior art.
  • FIG. 3 shows a schematic diagram of a method for restoring a stream in a central controller in the prior art.
  • FIG. 4 is a schematic diagram of an implementation manner of a scenario applied according to an embodiment of the present application.
  • FIG. 5 is an interaction flowchart of a network flow measurement method according to an embodiment of the present application.
  • FIG. 6 is a schematic diagram of a data structure of a network flow measurement method according to the present application.
  • FIG. 7 is a schematic diagram of a method for performing a network traffic measurement by a control plane device according to an embodiment of the present application.
  • FIG. 8 is a schematic diagram of an algorithm for a control plane device to extract a large stream from a stack according to an embodiment of the present application.
  • FIG. 9 is a schematic block diagram of a network measurement device according to an embodiment of the present application.
  • FIG. 10 is a schematic block diagram of a control plane device according to an embodiment of the present application.
  • FIG. 11 is a schematic block diagram of a network measurement device according to another embodiment of the present application.
  • FIG. 12 is a schematic block diagram of a control plane device according to another embodiment of the present application.
  • Count-minsketch is a sub-linear spatial data structure.
  • FIG. 1 shows a schematic diagram of a CMS structure.
  • CMS is composed of a two-dimensional array.
  • Each element of the array represents a count value, that is, the count value [1,1], ..., the count value [d, w].
  • Each element of the array is initialized to 0, and d independent hash functions are evenly and randomly selected.
  • the CMS uses a fixed d number of hash tables, each of which has a width of w. For each packet obtained, multiple hash operations are performed based on the key of the packet. In each hash table, the hash value is indexed to a specific hash bucket position, and the size of the recorded packets is accumulated.
  • the same hash functions are used to locate multiple buckets, and the bucket with the smallest value is determined from the buckets, and the value is an estimate of the size of the stream f.
  • the theory can prove that choosing a specific w, d can achieve quite high multi-stream query accuracy.
  • Flow data Generally refers to the packet flow with the same five-tuple (including source IP, destination IP, protocol number, source port, and destination port).
  • Stream entropy Calculation based on the frequency distribution of the stream.
  • K f hash functions measure whether it is a new data flow or an old data flow in a Bloom Filter (Flow Filter table).
  • count values in the Flow Filter table are all 1, it means that the old data flow is measured; if the count values in the Flow Filter table are 0, it means that the new data flow is measured, and the count value at the corresponding position is Set from 0 to 1.
  • K C hash functions are used to point to the index records.
  • FlowCnt can indicate the count value of the number of messages
  • PktCnt can indicate the length of the message or the number of messages. For example, as shown in FIG. 2, for the message a, the FlowCnt value is 1 and the PktCnt value is size (a).
  • the sensors deployed in the network device periodically send the recorded Flow Counter table to the central controller.
  • FlowRadar performs decoding algorithms to recover all flows.
  • FIG. 3 is a schematic diagram of a method for a central controller to recover a stream.
  • the key of the flow, PacketCnt is taken as the size of the flow; then K c hash is performed on the key, and the remaining positions corresponding to Cnt and PktCnt are subtracted. Based on this "peel onion" style, all flow record recovery can be completed.
  • the central controller FlowRadar decodes and cannot find an item with a FlowCnt value of 1, the flow cannot be recovered based on the above algorithm, resulting in decoding failure.
  • the network measurement function implemented by FlowRadar is mainly based on the flow information. Based on this, it is possible to query the number of flows, the flow mutation, and the flow distribution. But for common detection functions such as high flow detection, it does not support it well. Therefore, the above-mentioned flow detection method has poor versatility.
  • Requirement 1 High performance, that is, information extraction and statistics must be performed based on each message.
  • Requirement 2 Generality, that is, there are many aspects of measurement data requirements for network flow.
  • the CMS exemplified above can easily query the query "whether the size of the stream f exceeds the threshold T", but for the query "Find all streams exceeding the threshold T", it is necessary to enumerate all possible streams. And enumerating massive streams is computationally unacceptable.
  • sketch-based measurement technology usually requires different measurement algorithms for different information to be measured.
  • multiple sketch-based algorithms must be deployed simultaneously. The simultaneous running of multiple sketch-based algorithms will bring more processing and storage resource overhead.
  • this application proposes a method and device for network measurement.
  • an algorithm in the system can be used to achieve a variety of target flows such as large flows, arbitrary flows, and sudden flows. Measurement, and is not affected by the flow distribution, using a single sensor to support a wider range of network flow measurement applications.
  • FIG. 4 is a schematic diagram of an application scenario according to an embodiment of the present application.
  • the system may include a control plane device 101 and a plurality of nodes (for example, a network measurement device) 102.
  • the control plane device 101 is configured to receive measurement data sent by multiple nodes 102 in the system, such as receiving first data sent by multiple nodes 102.
  • the node 102 may be a device having a processing function, and may include, for example, any computing device known in the current technology, such as a server, a desktop computer, and the like.
  • the node 102 may include a memory and a processor.
  • Memory can be used to store program code, such as operating systems and other applications.
  • the processor can be used to call the program code stored in the memory to achieve the corresponding functions of the node.
  • the processor and memory included in the node may be implemented by a chip, which is not specifically limited here.
  • the system shown in FIG. 4 may have a distributed data plane and a centralized control plane.
  • the control plane device 101 can be regarded as a centralized control plane.
  • An operating system and other applications can be installed at the nodes.
  • a collection module may be deployed on each of the nodes 102.
  • the acquisition module runs a measurement algorithm.
  • the acquisition module obtains the message information from the forwarding module, executes the measurement algorithm update calculation, and the result is stored in the data structure.
  • the measurement algorithm may be a multi-sketch algorithm.
  • the node 102 may report the measurement data to the control plane device 101 periodically.
  • the acquisition module periodically reports this data structure to the control plane.
  • a data integration module can be included to collect data structures on all devices and process them to obtain classified data streams.
  • bit-level probability distributions for large flows and flow keywords can be obtained, and the output obtains large flow tables, small flow residuals, and bit-level flow distribution data. Based on these data, it supports many stream statistical queries, such as large stream query, stream size query, stream mutation query, stream number query, stream distribution query, etc.
  • FIG. 5 is a schematic flowchart of a network flow measurement method according to an embodiment of the present application.
  • the network measurement device may be any one of a plurality of measurement devices in the system, for example, it may be any node in FIG. 4.
  • the system may include multiple network measurement devices and a control plane device.
  • the control plane device processes the measurement data provided by the multiple network measurement devices, so as to obtain classified stream data.
  • the network measurement device measures first data, where the first data includes a first type of data structure, and the first type of data structure includes first measurement information, the first measurement information, and a bit of a key of the stream. correspond.
  • the first type of data structure may include first measurement information of at least one bit of a key of the corresponding stream.
  • the first type of data structure may include first measurement information of a part of bits corresponding to a key of a stream, or the first type of data structure may also include first measurement information of each bit of a key of a corresponding stream.
  • the first data is a data structure constructed based on the bit information of the key.
  • the first type of data structure included in the first data in one example may include the first measurement information of the first bit of the key of the corresponding stream; in one example, the first The first type of data structure included in the data may include first measurement information of each of the 3 bits of the key of the corresponding stream. This embodiment of the present application does not limit this.
  • the first data may further include a second type of data structure, and the second type of data structure includes measurement information of the keywords of the flow, that is, the measurement information recorded in the second type of data structure is based on the keywords of the flow.
  • the first data further includes a second type of data structure, and the second type of data structure includes second measurement information of the flow, and the second measurement information corresponds to a keyword of the flow.
  • first measurement information and the second measurement information may include a packet length of a flow, a number of flows, and parameters related to flow attributes.
  • first and second only refer to differences, which are not limited in this application.
  • the first measurement information is recorded based on the bits of the corresponding stream key
  • the second measurement information is recorded based on the entire stream key
  • the first data includes a first data structure and a second data structure, where the first data structure may include first measurement information corresponding to each bit of the keyword, and the second data
  • the data structure may include the second measurement information corresponding to the keywords of the stream for example.
  • the data structure shown in FIG. 6 may be a multi-sketch algorithm.
  • the flow ID for example, the flow key flowkey
  • L + 1 sketch data structures are constructed.
  • Each sketch data structure corresponds to each level.
  • Each sketch data structure can be a data cell matrix with r rows and c columns, where a small cell represents a data cell.
  • Each sketch hashes the keywords of the message r times, and each hash function independently and uniformly maps the keywords to the [1..c] space.
  • V i, j [k] is defined to represent the value of the i-th row and the j-th column in the data cell matrix in the k-th sketch
  • multi-sketch is the flow shown in FIG. 6.
  • the first type of data structure is K first data unit matrices, where each first data unit matrix is configured to record the first measurement according to a value of one bit of a key of the stream Information;
  • the second type of data structure is a second data unit matrix, the second data unit matrix is used to record the second measurement information, and K is a positive integer greater than or equal to 1.
  • the first type of data structure is K first data unit matrices, where one first data unit matrix is used to record one-bit measurement information of a key of the corresponding stream;
  • the second type The data structure is a second data unit matrix, and the second data unit matrix is used to record measurement information of keywords of the stream.
  • the first type of data structure can be regarded as a data structure of level 1 to level L, which is used to record the one-bit first measurement information of the key of the corresponding stream;
  • the second type of data structure can be regarded as leve 0
  • the data structure is used to record the second measurement information of the keywords of all streams. That is, the first type of data structure is a matrix of K first data units, and the second type of data structure is a second data unit matrix.
  • the K first data unit matrices are also L data structures.
  • the first measurement information and the second measurement information may be statistical values.
  • M ij in the second data unit matrix of the network measurement device records the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column, where i is a positive integer less than or equal to r , Where j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message; when the value of the Nth bit of the first stream keyword is a preset value , M ij of the Nth data unit matrix of the measurement device in the second type of data structure records the statistical value of the first stream, the N is less than or equal to the K, and the K and the N Both are positive integers.
  • the measurement device when the value of the Nth bit of the first stream keyword is 1, the measurement device records the M ij of the Nth data unit matrix in the second type of data structure.
  • the N is less than or equal to the K, and the K and the N are both positive integers.
  • the measurement device when the value of the Nth bit of the first stream keyword is 0, the measurement device records the M ij of the Nth data unit matrix in the second type of data structure.
  • the N is less than or equal to the K, and the K and the N are both positive integers.
  • the statistics of all keywords can be recorded in sketch 0.
  • sketch-1 to sketch-L will count the value of each bit in the keyword as a preset value.
  • the preset value may be 1 or the preset value may also be 0, which is not limited in the embodiment of the present application.
  • f [k] represents the value of the k-th bit of the stream ID.
  • the data unit records for level are as follows:
  • the number of flow packets is exemplified.
  • the keyword 1 shown in FIG. 6 is 0101, and the hash function is indexed to the first row and the second column of the level 0 data unit by the h 1 hash function; the hash function is indexed to the level 0 data unit by the h 2 hash function. Row 2 and Column 7.
  • the keywords are counted bit by bit. For example, if the value of the first bit of the keyword 1 is not 1, the value of V 1,2 [1] and the value of V 2,7 [1] are not operated.
  • the value of line 1 and line 2 in level 1 is not operated;
  • the value of the second bit of keyword 1 is 1, the value of V 1,2 [2] and the value of V 2,7 [2] increase 1, that is, operate on the value of line 1 and line 2 in level 2;
  • the value of the third bit of keyword 1 is not 1, then the value of V 1,2 [3] and V 2,7 [3] Operation on the value of line 1, that is, the value on line 1 and line 2 of level 3 is not operated;
  • the value of the fourth bit of keyword 1 is 1, then the value of V 1,2 [4] is the same as V 2,7 [
  • the value of 4] is increased by 1, that is, the value of line 1 and line 2 in level 4 is operated.
  • a length value of a statistical flow packet is used as an example.
  • the key 1 shown in FIG. 6 is 0101, and the hash function is indexed to the first row and the second column of the level0 data unit by the h1 hash function; the h2 hash function is used to index the hash function to the level 0 data unit. Row 2, column 7.
  • the keywords are counted bit by bit. For example, if the value of the first bit of the keyword 1 is not 1, the value of V 1,2 [1] and the value of V 2,7 [1] are not operated.
  • the value of line 1 and line 2 in level 1 is not operated; the value of the second bit of keyword 1 is 1, the value of V 1,2 [2] and the value of V 2,7 [2] increase Keyword 1 corresponds to the packet length value, that is, the value of line 1 and line 2 in level 2 is operated; the value of the third bit of keyword 1 is not 1, then the value of V 1,2 [3] is not And the value of V 2,7 [3], that is, the value of line 1 and line 2 in level 3 is not operated; the value of the fourth bit of keyword 1 is 1, then V 1,2 [4] If the value of the value and the value of V 2,7 [4] is increased by the value of the message length corresponding to the keyword 1, the value of the first line and the second line in level 4 is operated.
  • keyword 1 is similar to the above process, and will not be repeated here.
  • a description is made by taking, for example, sketch-1 to sketch-L, statistics of the value of each bit in the keyword as '0'.
  • f [k] represents the value of the k-th bit of the stream ID
  • the data unit records for level are as follows:
  • FIG. 6 illustrates a process in which the network measurement device measures the first data, and records the statistical result into a data unit.
  • the network measurement device measures the first data, and records the statistical result into a data unit.
  • the network measurement device sends the first data to the control plane device.
  • control plane device receives the first data sent by the network measurement device, and obtains a statistical result according to the first data.
  • the statistical result may include a list of large streams in the first data, or data of a residual stream other than the large stream in the first data, or a frequency of each bit of stream data in the first data. At least one of the distributions.
  • it supports large stream query, stream size query, stream mutation query, stream number query, stream distribution query, etc.
  • the first data may include a first type of data structure and a second type of data structure, and the first type of data structure may include measurement information of bits corresponding to the keyword.
  • the first type of data structure may include measurement information of at least one bit of the corresponding keyword, or the first type of data structure may include measurement information of each bit of the corresponding keyword.
  • the first data may further include a second type of data structure, and the second type of data structure includes measurement information of a keyword corresponding to the stream.
  • M ij in the second type of data structure is used to record the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column, and i is a positive integer less than or equal to r , Where j is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the message length;
  • the first type of data structure is K first data unit matrices, where M ij in the Nth data unit matrix is used when the value of the Nth bit of the first stream keyword is a preset value, Record the statistics of the first stream, where N is less than or equal to K, and K and N are positive integers.
  • the first type of data structure is K second data unit matrices, where M ij in the Nth data unit matrix is used when the value of the Nth bit of the first stream key is 1, Record the statistics of the first stream, where N is less than or equal to K, and K and N are positive integers.
  • the first type of data structure is K second data unit matrices, where M ij in the Nth data unit matrix is used when the value of the Nth bit of the first stream key is 0, Record the statistics of the first stream, where N is less than or equal to K, and K and N are positive integers.
  • the second type of data structure may be a level 0 data structure.
  • the data structure of the first type may be data units of level 1 to level 4.
  • M ij can be expressed as the first row, second column, or second row and seventh column in each data structure, that is, the information of the row and column corresponding to the position where the keyword 1 is recorded in the level 0 data unit.
  • the first data includes K + 1 element matrices, that is, includes a second data element matrix and K first data element matrices, where the number of the first data element matrices may be related to the number of bits of the stream key.
  • the first data may include 3 first data unit matrices.
  • K + 1 data unit matrix streams with the same hash conflict are recorded in the same row and column.
  • the K + 1 data unit group is called a stack, and (i, j) represents the i-th Row j column stack, so there are r * c stack stacks.
  • n i, j be the number of streams hashed to (i, j), and n be the number of all streams observed, then
  • Theorem 1 For any stack (i, j) and level, if stack (i, j) does not include large flow, then
  • Theorem 1 It is required that there is no large flow in a stack.
  • the basic idea of the embodiment of the present application is that if all large flows are separated from the multi-level, the sketch results of the remaining flows will obey the Gaussian distribution.
  • the large stream can be separated bit by bit, and by checking whether the remaining sketch conforms to the Gaussian distribution, it can be determined whether the iteration is completed. And this basic idea, you can use Hypothesis 2 to operate on each stack to achieve all the flow out.
  • the obtaining, by the control plane device, a statistical result according to the first data includes:
  • the control plane device Obtaining, by the control plane device, a first set and updated first data according to a distribution characteristic on each bit of the keyword, the first set including r * c stacks included from the first data A large stream extracted in the stream, the updated first data includes all data units corresponding to the r * c stack stack except the residual stream of the first set, in the r * c stacks Any one of the stacks is a data unit group composed of data units of the same row and column in the K + 1 data unit matrix.
  • control plane device obtains the first set and the updated first data according to the distribution characteristics (such as probability distribution) on each bit of the keyword, and all stack stacks included in the first data by the control plane device are r *
  • distribution characteristics such as probability distribution
  • all stack stacks included in the first data by the control plane device are r *
  • the operations of extracting large streams in c stack stacks are performed, and the large streams of all the extracted stacks are recorded in the first set, and the updated first data includes the r * c stack stacks except the first All data units corresponding to the set of residual streams.
  • control plane device may perform an extraction of a large stream based on all stacks and update it once First data.
  • the control plane device extracts a large stream from the first stack.
  • the first stack is any one of r * c stack stacks.
  • the first set may include the large stream extracted from the first stack, and the updated first data. It may include all data units corresponding to the r * c stack stacks except the residual stream extracted from the first stack in the first stack.
  • the first set may include a large stream in the first data
  • the updated first data may include a residual stream other than the large stream in the first data
  • the updated first data satisfies a normal distribution, it is determined that the first set is a large stream.
  • a stack does not include a large flow, its probability distribution characteristics obey Gaussian distribution. That is, if the updated first data satisfies a normal distribution, it means that the updated first data does not include a large flow, which means that the disjuncted large flow included in the first set is a large flow.
  • a large stream is extracted from the r * c stacks included in the first data.
  • the updated first data does not satisfy the normal distribution, it means that the undissolved large stream may also be included in the updated first data, and then the large data is continuously extracted from all stacks in the first data. flow. The extracted large stream is recorded in the first set.
  • the updated first data satisfies a normal distribution, it can be explained that the large stream in the first data has been completely extracted, and the first set is a large stream at this time.
  • a large stream is extracted from the r * c stacks included in the first data after adjusting the threshold, where The threshold value indicates a ratio of a large flow rate to a total flow rate or a ratio of a large flow rate to a total flow rate.
  • the threshold can be ⁇ , ⁇ represents a parameter for judging a large flow. If the frequency or size of a flow f> ⁇ N, the flow is said to be a large flow, where N is the frequency of all the flows in an observation period. Degrees or sizes.
  • any stack does not include a large flow, its probability distribution characteristics obey Gaussian distribution.
  • the updated first data does not satisfy the normal distribution, it means that the updated first data also includes a large stream.
  • extracting a large stream from all the stacks included in the first data may include the following process.
  • the control plane device obtains the corresponding r * c stack stacks.
  • a probability that the value of the bit is 1; a third set is determined according to the probability, and the third set includes candidate keywords of the large stream; and a keyword of the large stream is determined according to the third set.
  • determining the keywords of the high flow according to the third set includes:
  • a universal measurement method is implemented to support a large stream, an arbitrary stream size, and a sudden stream. It can measure multiple kinds of target traffic and is not affected by the flow distribution. It realizes the application of using a single sensor to support wider network flow measurement.
  • FIG. 7 is a schematic diagram of a method for performing a network traffic measurement by a control panel device according to an embodiment of the present application.
  • the network traffic measurement method in FIG. 7 may be performed by a control plane device in the system.
  • a control plane device in the system.
  • it may be the control plane device 101 in FIG. 4.
  • the method includes:
  • control plane device calculates the mean and variance in each sketch (1 to L).
  • stack (i, j) is also deleted, and then the bit-level distribution is recalculated.
  • S350 Determine whether the residual stream after deleting the large stream satisfies a normal distribution. If the normal distribution is satisfied, S370 is executed to output the large stream set F, the data structure of the residual stream, and the bit-level distribution N. If the normal distribution is not satisfied, S360 is executed.
  • the size of the sequence numbers of the above processes does not mean the order of execution.
  • the execution order of each process should be determined by its function and internal logic, and should not deal with the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • FIG. 7 depicts a flowchart of the above steps, and a pseudo-code implementation of the algorithm shown in FIG. 7 will be described below.
  • Multi-level sketch S ⁇ V i, j [k]
  • F ' F' ⁇ ExtractLagerFlows ( ⁇ , (i, j), S, ⁇ N (p [k], ⁇ 2 [k]) ⁇ );
  • the above code is a pseudo code that can execute the flowchart shown in FIG. 7.
  • FIG. 8 is a schematic diagram of an algorithm for extracting a large stream from a stack according to an embodiment of the present application. According to the flow shown in FIG. 8, the control plane device can extract a large stream from the r * c stacks included in the first data to the first set.
  • Ri , j [k] is the frequency value recorded in the k-th level sketch in the stack (i, j) before the large stream is extracted and the frequency in level-0.
  • S420 Use the probability of each bit to construct a stream key of the candidate big stream, and use a hash function to check the correctness.
  • ⁇ is 0.99
  • the value of the k-th bit is estimated to be 1 or 0. If neither is satisfied, the k-th bit is recorded as *, which means that the value of the k-th bit can be 1 or 0.
  • the frequency average of each bit is used to obtain the frequency of the large stream f.
  • a flowkey with a certainty probability of 50% of the number of bits greater than 90% is filtered and the others are discarded.
  • each sketch has r hash tables.
  • control plane device can obtain classified stream data.
  • a large flow table can be obtained to record the flowkey and corresponding frequency of the large flow; the data structure record of the residual flow is similar to the structure of a multi-level sketch as shown in FIG. 6, except that the influence of the large flow is removed; Bit-level distribution of Ri , j [k] .
  • the output can be directly judged in the large flow table.
  • the flow mutation query records two periods of data, queries in a large flow table, and performs a per-flow-frequency query in another period of data.
  • the flow distribution query uses the residual sketch data and the MRAC algorithm to obtain the flow distribution. Further, the flow distribution data can be used to calculate the flow entropy entropy.
  • n (p [k] (1-p [k]) c / ( ⁇ 2 [k])
  • the size of the sequence numbers of the above processes does not mean the order of execution.
  • the execution order of each process should be determined by its function and internal logic, and should not deal with the embodiments of the present application.
  • the implementation process constitutes any limitation.
  • the network measurement device can perform measurement based on the bit level of the flow key and record the measurement information; the control plane device obtains the measurement information from the network device. Statistics results, so as to realize the query that can flexibly support multiple streams.
  • the network measurement device and the control plane device in the embodiments of the present application can execute the various methods of the foregoing embodiments of the present application, that is, the specific working processes of the following various products, refer to the corresponding processes in the foregoing method embodiments.
  • FIG. 9 shows a schematic block diagram of a network measurement device 700 according to an embodiment of the present application (the network measurement device in FIG. 9 may be any node in FIG. 4).
  • the network measurement device 700 may correspond to the network measurement device in each method embodiment, and may have any function of the network measurement device in the method.
  • the network measurement device 700 includes a communication module 710 and a processing module 720.
  • the processing module 720 is configured to measure first data, where the first data includes a first type of data structure, and the first type of data structure includes first measurement information of a stream, and the first measurement information and a key of the stream The bits of the word correspond.
  • the communication module 710 is configured to send the first data to the control plane device.
  • the network measurement device performs flow data measurement by using a bit-level first-level data structure corresponding to a keyword of a flow included in the first data, and reports measurement information based on the first data record to the control plane.
  • the control plane device obtains a statistical result according to the probability distribution characteristic of the first data. Therefore, a universal measurement method can be used to support the measurement of multiple target flows such as large flows, arbitrary flows, and sudden flows, and it is not affected by the flow distribution.
  • a single sensor is used to support a wide range of network flow measurements. application.
  • the first type of data structure includes measurement information of at least one bit of a key of a corresponding stream.
  • the first data further includes a second type of data structure, and the second type of data structure includes second measurement information of the flow, and the second measurement information corresponds to a keyword of the flow.
  • the first measurement information corresponds to each bit of a key of the stream.
  • the first type of data structure includes measurement information corresponding to each bit of the keyword.
  • the first type of data structure is K first data unit matrices, where each first data unit matrix is configured to record the first measurement according to a value of one bit of a key of the stream Information;
  • the second type of data structure is a second data unit matrix, the second data unit matrix is used to record the second measurement information, and K is a positive integer greater than or equal to 1.
  • each of the K first data unit matrices corresponds to one bit of a key of a stream.
  • the four first data unit matrices correspond to the key of the stream, that is, the first bit of the key corresponds to the first first data unit matrix, and the second bit of the key Corresponds to the second first data unit matrix, the third bit corresponds to the third first data unit matrix, and the fourth bit corresponds to the fourth first data unit matrix.
  • the processing module 720 is specifically configured to record the second measurement information of the first stream in M ij in the second data unit matrix, where M ij represents a data unit in an i-th row and a j-th column , I is a positive integer less than or equal to r, j is a positive integer less than or equal to c, and the second measurement information includes at least one of the number of packets or the length of the packets in the first stream ;
  • M ij of the Nth data unit matrix in the first type of data structure records the first of the first stream Measurement information
  • the first measurement information includes at least one of a number of packets or a length of a packet in the first stream
  • the N is less than or equal to the K
  • the K and the N are positive integers .
  • the first measurement information and the second measurement information may be statistical values.
  • the processing module 720 is specifically configured to, when the value of the Nth bit of the first stream keyword is 1, the M ij record of the Nth data unit matrix in the first type of data structure For the statistical value of the first stream, the N is less than or equal to the K, and the K and the N are both positive integers.
  • the processing module 720 is specifically configured to, when the value of the Nth bit of the first stream keyword is 0, the M ij record of the Nth data unit matrix in the first type of data structure For the statistical value of the first stream, the N is less than or equal to the K, and the K and the N are both positive integers.
  • mapping relationship exists between the first stream and M ij , and the mapping relationship is:
  • h i () represents the i-th hash function
  • the key represents a keyword corresponding to the first stream.
  • the processing module included in the network measurement device may be a multi-sketch processing unit, and the multi-sketch may record measurement information based on the data structure shown in FIG. 6.
  • the communication module can be, for example, a function implemented on a software device, or an algorithm unit on a hardware device, which is used to send the data structure of the multi-sketch record to the control plane.
  • FIG. 10 shows a schematic block diagram of a control plane device 800 according to an embodiment of the present application (the control plane device 800 in FIG. 8 may be the control plane device in FIG. 4).
  • the control plane device 800 may correspond to the control plane device in each method embodiment, and may have any function of the control plane device in the method.
  • control plane device 800 may include a communication module 810 and a processing module 820.
  • the communication module 810 is configured to obtain first data reported by the network measurement device, where the first data includes a first type of data structure, and the first type of data structure includes first measurement information of a stream, and the first measurement The information corresponds to the bits of the key of the stream;
  • the processing module 820 is configured to obtain a statistical result according to the first data.
  • the control plane device receives the measurement information reported by the network measurement device, and the network measurement device measures the stream data by using the first type data structure of the bit level corresponding to the key of the stream included in the first data, and The measurement information based on the first data record is reported to the control plane device, and the control plane device obtains a statistical result according to the probability distribution characteristic of the first data. Therefore, a universal measurement method can be used to support the measurement of multiple target flows such as large flows, arbitrary flows, and sudden flows, and it is not affected by the flow distribution. A single sensor is used to support a wide range of network flow measurements. application.
  • the first type of data structure includes measurement information of at least one bit of a key of a corresponding stream.
  • the first data further includes a second type of data structure, and the second type of data structure includes second measurement information of the flow, and the second measurement information corresponds to a keyword of the flow.
  • the first type of data structure includes measurement information corresponding to each bit of the keyword.
  • the first measurement information corresponds to each bit of a key of the stream.
  • the first type of data structure is a matrix of K first data units, where each first data unit is configured to record the first measurement information according to a value of one bit of a key of the stream
  • the second type of data structure is a second data unit matrix, the second data unit matrix is used to record the second measurement information, and K is a positive integer greater than or equal to 1.
  • each of the K first data unit matrices corresponds to one bit of a key of a stream.
  • the four first data unit matrices correspond to the key of the stream, that is, the first bit of the key corresponds to the first first data unit matrix, and the second bit of the key Corresponds to the second first data unit matrix, the third bit corresponds to the third first data unit matrix, and the fourth bit corresponds to the fourth first data unit matrix.
  • M ij in the second type of data structure is used to record the second measurement information of the first stream, where M ij represents a data unit in the i-th row and the j-th column, and i is less than or equal to a positive integer of r, j being a positive integer less than or equal to c, and the second measurement information including at least one of the number of packets or the length of the packets in the first stream;
  • the first type of data structure is K first data unit matrices, where M ij in the Nth data unit matrix is used when the value of the Nth bit of the first stream keyword is a preset value, Record the first measurement information of the first stream, the second measurement information includes at least one of the number of messages or the message length in the first stream, and the N is less than or equal to the K, so Let N be a positive integer.
  • the first measurement information and the second measurement information may be statistical values.
  • M ij in the second type of data structure is used to record the statistical value of the first stream, where M ij represents a data unit in the i-th row and the j-th column, and i is a positive integer less than or equal to r , J is a positive integer less than or equal to c, and the statistical value includes at least one of the number of messages or the length of the message;
  • the first type of data structure is a matrix of K first data units, where M ij in the N data unit matrix is used to record the statistical value of the first stream when the value of the Nth bit of the first stream keyword is a preset value, where N is less than or equal to the K, Both K and N are positive integers.
  • the first type of data structure is K first data unit matrices, where M ij in the Nth data unit matrix is used when the value of the Nth bit of the first stream key is 1, Record the statistics of the first stream, where N is less than or equal to K, and K and N are positive integers.
  • the first type of data structure is K first data unit matrices, where M ij in the Nth data unit matrix is used when the value of the Nth bit of the first stream key is 0, Record the statistics of the first stream, where N is less than or equal to K, and K and N are positive integers.
  • mapping relationship exists between the first stream and M ij , and the mapping relationship is:
  • h i () represents the i-th hash function
  • the key represents a keyword corresponding to the first stream.
  • the processing module 820 is specifically configured to obtain a first set and updated first data according to a distribution characteristic on each bit of the keyword, where the first set includes data from the first data A large stream extracted from the r * c stack stacks, and the updated first data includes all data units corresponding to the r * c stack stacks except the residual stream of the first set, Any one of the r * c stacks is a data unit group composed of data units of the same row and column in the K + 1 data unit matrix.
  • the processing module 820 is specifically configured to extract a large stream from the first stack of the r * c stack stacks, and the first set includes the large stream extracted from the first stack.
  • the updated first data includes all data units corresponding to the r * c stack stacks except for the residual stream extracted from the first stack.
  • the processing module 820 is further configured to determine that the first set includes all large flows in the first data.
  • the processing module 820 is further configured to extract a large stream from the r * c stacks included in the first data .
  • the processing module 820 is specifically configured to extract a large stream from the r * c stack stacks included in the first data after adjusting a threshold, where:
  • the threshold value indicates a ratio of a large flow rate to a total flow rate or a ratio of a large flow rate to a total flow rate.
  • the processing module 820 is specifically configured to obtain a probability that the value of each bit in the r * c stacks is 1.
  • the third set is determined according to the probability, and the third set includes the third set.
  • the processing module 820 is specifically configured to determine K frequencies based on a ratio of each bit of the first type data structure to the same stack in the second type data structure, and calculate the third set The frequency of at least one keyword included; filtering the at least one keyword according to the certainty probability of each bit to determine the keywords of the large stream; frequency of keywords of the large stream according to a hash function check.
  • the statistical result includes a large stream list in the first data, or data of a residual stream other than the large stream in the first data, or a bit stream data of each bit in the first data. At least one of the frequency distributions.
  • the communication module 810 is further configured to receive a traffic query request by the control plane device;
  • the processing module 820 is further configured to send a query result according to the traffic query request.
  • FIG. 11 is a structural block diagram of a network measurement device according to an embodiment of the present application.
  • the network measurement device includes a processor 801, a memory 802, and a transceiver 803.
  • the processor 801 may be configured to process stream data, control a network measurement device, execute a software program, process data of the software program, and the like.
  • the memory 802 is mainly used to store software programs and data.
  • the processor 801 processes the stream data to be transmitted and outputs the stream data to the transceiver 803.
  • the transceiver 803 receives the stream data and outputs the data to the processor, and the processor processes the stream data.
  • FIG. 8 only one memory and processor are shown in FIG. 8. In an actual network equipment product, there may be one or more processors and one or more memories.
  • the memory may also be referred to as a storage medium or a storage device.
  • the memory may be set independently of the processor or integrated with the processor, which is not limited in the embodiment of the present application.
  • a circuit having a transceiving function may be regarded as a transceiver 803 of a network device, and a processor having a processing function may be regarded as a processing module 720 of a network measuring device.
  • the transceiver may also be referred to as a transceiver module 710, a transceiver, a transceiver device, and the like.
  • the processing module may also be called a processor, a processing board, a processing module, a processing device, and the like.
  • the device for implementing the receiving function in the transceiver 803 may be regarded as a receiving unit, and the device for implementing the transmitting function in the transceiver 803 may be regarded as a transmitting unit, that is, the transceiver 803 includes a receiving unit and a transmitting unit.
  • the receiving unit may also be called a receiver, a receiver, or a receiving circuit.
  • the transmitting unit may also be called a transmitter, a transmitter, or a transmitting circuit.
  • the processor 801, the memory 802, and the transceiver 803 communicate with each other through an internal connection path, and transfer control and / or data signals
  • the method disclosed in the foregoing embodiment of the present application may be applied to the processor 801, or implemented by the processor 801.
  • the processor 801 may be an integrated circuit chip and has a signal processing capability.
  • each step of the above method may be completed by using hardware integrated logic circuits or instructions in the form of software in the processor 801.
  • the processor described in the embodiments of the present application may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), and a ready-made programmable gate array (field programmable gate array). , FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the steps of the method disclosed in combination with the embodiments of the present application may be directly implemented by a hardware decoding processor, or may be performed by using a combination of hardware and software modules in the decoding processor.
  • Software modules can be located in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc.
  • Storage media The storage medium is located in a memory, and the processor reads the instructions in the memory and completes the steps of the foregoing method in combination with its hardware.
  • the memory 802 may store instructions for executing a method performed by a network measurement device in the method shown in FIG. 5.
  • the processor 801 may execute instructions stored in the memory 802 in combination with other hardware (for example, the transceiver 803) to complete the steps performed by the network measurement device in the method shown in FIG. 5.
  • other hardware for example, the transceiver 803
  • An embodiment of the present application further provides a chip, and the chip includes a transceiver unit and a processing unit.
  • the transceiver unit may be an input / output circuit or a communication interface;
  • the processing unit is a processor or a microprocessor or an integrated circuit integrated on the chip.
  • the chip can execute the method of the network measurement device in the method embodiment shown in FIG. 5.
  • the embodiment of the present application further provides a computer program product containing instructions, and when the instructions are executed, the method of the network measurement device in the method embodiment shown in FIG. 4 is executed.
  • An embodiment of the present application further provides a computer-readable storage medium having instructions stored thereon, which are executed when the method of the network measurement device in the method embodiment shown in FIG. 5 is executed.
  • FIG. 12 is a structural block diagram of a control plane device according to an embodiment of the present application.
  • the control plane device includes a processor 901, a memory 902, and a transceiver 903.
  • the processor 901 may be configured to process stream data, control a network measurement device, execute a software program, process data of the software program, and the like.
  • the memory 902 is mainly used to store software programs and data.
  • the processor 901 processes the stream data to be transmitted, and outputs it to the transceiver 903.
  • the transceiver 903 receives the stream data and outputs the data to the processor, and the processor processes the stream data.
  • FIG. 12 only one memory and processor are shown in FIG. 12. In an actual network equipment product, there may be one or more processors and one or more memories.
  • the memory may also be referred to as a storage medium or a storage device.
  • the memory may be set independently of the processor or integrated with the processor, which is not limited in the embodiment of the present application.
  • a circuit having a transceiving function may be regarded as a transceiver 903 of a network device, and a processor having a processing function may be regarded as a processing module 820 of a control plane device.
  • the transceiver may also be referred to as a communication module 810, a transceiver, a transceiver device, and the like.
  • the processing module may also be called a processor, a processing board, a processing module, a processing device, and the like.
  • a device for implementing a receiving function in the transceiver 903 may be regarded as a receiving unit, and a device for implementing a transmitting function in the transceiver 903 may be regarded as a transmitting unit, that is, the transceiver 903 includes a receiving unit and a transmitting unit.
  • the receiving unit may also be called a receiver, a receiver, or a receiving circuit.
  • the transmitting unit may also be called a transmitter, a transmitter, or a transmitting circuit.
  • the processor 901, the memory 902, and the transceiver 903 communicate with each other through an internal connection path, and transfer control and / or data signals
  • the methods disclosed in the foregoing embodiments of the present application may be applied to the processor 901, or implemented by the processor 901.
  • the processor 901 may be an integrated circuit chip and has a signal processing capability.
  • each step of the above method may be completed by using hardware integrated logic circuits or instructions in the form of software in the processor 901.
  • the processor described in the embodiments of the present application may be a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), and a ready-made programmable gate array (field programmable gate array). , FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
  • DSP digital signal processor
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
  • the steps of the method disclosed in combination with the embodiments of the present application may be directly implemented by a hardware decoding processor, or may be performed by using a combination of hardware and software modules in the decoding processor.
  • Software modules can be located in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory or electrically erasable programmable memory, registers, etc.
  • Storage media The storage medium is located in a memory, and the processor reads the instructions in the memory and completes the steps of the foregoing method in combination with its hardware.
  • the memory 902 may store instructions for executing a method performed by a control plane device in the method shown in FIG. 5.
  • the processor 901 may execute the instructions stored in the memory 902 in combination with other hardware (such as the transceiver 903) to complete the steps performed by the control plane device in the method shown in FIG. description of.
  • An embodiment of the present application further provides a chip, and the chip includes a transceiver unit and a processing unit.
  • the transceiver unit may be an input / output circuit or a communication interface;
  • the processing unit is a processor or a microprocessor or an integrated circuit integrated on the chip.
  • the chip can execute the method for controlling a surface device in the method embodiment shown in FIG. 5.
  • the embodiment of the present application also provides a computer program product containing instructions, and when the instructions are executed, the method for controlling a surface device in the method embodiment shown in FIG. 5 is executed.
  • An embodiment of the present application further provides a computer-readable storage medium, which stores instructions thereon, and when the instructions are executed, the method for controlling a surface device in the method embodiment shown in FIG. 5 is executed.
  • the disclosed systems, devices, and methods may be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the unit is only a logical function division.
  • multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not implemented.
  • the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, which may be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the objective of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each of the units may exist separately physically, or two or more units may be integrated into one unit.
  • the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.
  • the technical solution of this application is essentially a part that contributes to the existing technology or a part of the technical solution can be embodied in the form of a software product.
  • the computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present application.
  • the aforementioned storage media include: U disks, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks or compact discs and other media that can store program codes .

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Environmental & Geological Engineering (AREA)
  • Computer Security & Cryptography (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Medical Informatics (AREA)
  • Evolutionary Computation (AREA)
  • Databases & Information Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Algebra (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Pure & Applied Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Computer Hardware Design (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本申请提供了一种网络流测量的方法、网络测量设备以及控制面设备，该方法适用于包括网络测量设备和控制面设备的系统中，该方法包括：该网络测量设备测量第一数据，该第一数据包括第一类数据结构，该第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应；该网络测量设备向该控制面设备发送该第一数据。基于上述技术方案，该网络测量设备可以基于流关键字的比特级进行测量记录测量信息，从而实现网络流测量的通用性。

Description

网络流测量的方法、网络测量设备以及控制面设备
本申请要求于2018年07月27日提交中国专利局、申请号为201810844967.6、申请名称为“网络流测量的方法、网络测量设备以及控制面设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及信息技术领域，更具体地，涉及网络流测量的方法、网络测量设备以及控制面设备。
背景技术
大数据、人工智能技术的发展推动网络智能化，实现网络智能化的基础是网络的全网全量的数据测量，而对于网络测量技术而言，需要满足两个关键的要求。第一、高性能，即需要做到基于每个报文进行信息提取和统计；第二、通用性，即对于网络流要求有多方面的测量数据时，能够采用少量的通用传感器以支撑众多的测量目标。
现有技术中基于概要技术的网络测量，针对不同的待测量信息，通常需要采用不同的测量算法。当一个实际的测量任务需要多种信息时，需要同时部署多个基于概要的算法。但是当多个基于概要的算法的同时运行，导致处理与存贮资源开销较大。因此，目前的网络流测量的方法存在通用性较差的问题。
发明内容
本申请提供一种网络流测量的方法、网络测量设备以及控制面设备，能够基于流关键字的比特级进行测量，记录测量信息，从而实现网络流测量的通用性。
第一方面，本申请实施例提供了一种网络流测量的方法，其特征在于，所述方法适用于包括网络测量设备和控制面设备的系统中，所述方法包括：所述网络测量设备测量第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括第一测量信息，所述第一测量信息与所述流的关键字的比特对应；所述网络测量设备向所述控制面设备发送所述第一数据。
在本申请实施例的技术方案中，网络测量设备通过第一数据包括的对应流的关键字的比特级的第一类数据结构进行流数据的测量，将基于第一数据记录的测量信息上报至控制面设备。从而实现了通过一种通用的测量方法，可以支持大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量的应用。
结合第一方面，在第一方面的某些实现方式中，所述第一类数据结构包括对应流的关键字的至少一个比特的测量信息。
例如，在本申请的实施例中，流对应的关键字为4比特，第一类数据结构可以包括对 应该流关键字的4个比特中的3比特或者2比特的测量信息，即第一类数据结构可以包括对应于关键字的部分比特的测量信息。
结合第一方面，在第一方面的某些实现方式中，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息。
在本申请实施例的技术方案中，第一数据还可以包括第二类数据结构，第二类数据结构中可以记录所有流的关键字，第一类数据结构和第二类数据结构构成网络测量设备对流数据进行测量的数据结构。
结合第一方面，在第一方面的某些实现方式中，所述第一类数据结构包括对应所述关键字的每个比特的测量信息。
例如，在本申请的实施例中，流对应的关键字为4比特，第一类数据结构可以包括对应该流关键字的4个比特的测量信息，即第一类数据结构可以包括对应于关键字的4个比特的测量信息。
结合第一方面，在第一方面的某些实现方式中，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
应理解，K个第一数据单元矩阵中的每一个数据单元矩阵与流的关键字的一个比特对应。
例如，流的关键字为4比特，则4个第一数据单元矩阵与流的关键字相对应，即关键字的第一比特与第一个第一数据单元矩阵对应，关键字的第二比特与第二个第一数据单元矩阵对应，第三比特与第三个第一数据单元矩阵对应，第四比特与第四个第一数据单元矩阵对应。
应理解，数据单元矩阵可以是包括r行c列的数据结构，其中，包括r*c个数据单元。
在本申请实施例的技术方案中，第一类数据结构包括的第一矩阵单元的数量与对应流的关键字的比特数相关，若对应流的关键字包括K个比特，则第一类数据结构可以包括1个至K个第一数据单元矩阵，其中，第一个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息，第K个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息；第二类数据单元结构包括一个第二数据单元矩阵，在第二数据单元矩阵中基于对应流的关键字记录测量信息，即将基于对应流的关键字的整体记录测量信息。
应理解，在本申请的实施例中，测量信息可以是报文数量，或报文长度，或与报文属性相关的参数。
结合第一方面，在第一方面的某些实现方式中，所述方法还包括：
所述网络测量设备在所述第二数据单元矩阵中的M ij记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；
在所述第一流关键字的第N比特的取值为预设值时，所述测量设备在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的所述第一测量信息，所述第一测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K， 所述K与所述N均为正整数。
结合第一方面，在第一方面的某些实现方式中，所述预设值为1。
结合第一方面，在第一方面的某些实现方式中，所述方法还包括：所述网络测量设备在所述第二数据单元矩阵中的M ij记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为预设值时，所述测量设备在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。例如，在一个示例中，所述网络测量设备在所述第二数据单元矩阵中的M ij记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为1时，所述测量设备在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，在一个示例中，所述网络测量设备在所述第二数据单元矩阵中的M ij记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为0时，所述测量设备在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
需要说明的是，第一类数据结构中基于对应流的关键字的比特记录第一流的统计值的数据单元的位置与第二类数据结构中基于对应流的关键字整体记录第一流的统计值的数据单元的位置相同。
例如，第二类数据结构即为第二数据单元矩阵，若第一流的关键字在第二数据单元矩阵中的第i行第j列中记录了统计值，则在K个第一数据单元矩阵的第i行第j列中记录了对应第一流关键字的比特的统计值。
结合第一方面，在第一方面的某些实现方式中，所述第一流与M ij存在映射关系，所述映射关系为：h i(key)＝j；其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
在本申请的实施例的技术方案中，第一流与记录第一流统计值的数据单元M ij之间存在着映射关系，即h i(key)＝j，h i(x)是第i个hash函数。此外M ij可以记录多条流的统计值。这多条流有共同的特征，即通过相同的哈希函数根据哈希值索引到数据单元矩阵中相同的数据单元。
第二方面，本申请实施例提供了一种网络流测量的方法，所述方法适用于包括网络测量设备和控制面设备的系统中，所述方法包括：所述控制面设备获取所述网络测量设备上报的第一数据，所述第一数据包括第一类数，所述第一类数据结构包括第一测量信息，所述第一测量信息与所述流的关键字的比特对应；所述控制面设备根据所述第一数据，得到统计结果。
在本申请的实施例中，控制面设备接收网络测量设备上报的测量信息，网络测量设备通过第一数据包括的对应流的关键字的比特级的第一类数据结构进行流数据的测量，将基 于第一数据记录的测量信息上报至控制面设备，控制面设备根据第一数据的概率分布特性，得到统计结果。从而实现了通过一种通用的测量方法，可以支持大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量的应用。
结合第二方面，在第二方面的某些实现方式中，所述第一类数据结构包括对应流的关键字的至少一个比特的测量信息。
例如，在本申请的实施例中，流对应的关键字为4比特，第一类数据结构可以包括对应该流关键字的4个比特中的3比特或者2比特的测量信息，即第一类数据结构可以包括对应于关键字的部分比特的测量信息。
结合第二方面，在第二方面的某些实现方式中，所述第一数据还包括第二类数据结构，所述第二类数据结构包括对应所述流的关键字的测量信息。
结合第二方面，在第二方面的某些实现方式中，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
在本申请实施例的技术方案中，第一数据还可以包括第二类数据结构，第二类数据结构中可以记录所有流的关键字，第一类数据结构和第二类数据结构构成网络测量设备对流数据进行测量的数据结构。
结合第二方面，在第二方面的某些实现方式中，所述第一测量信息与所述流的关键字的每个比特对应。
结合第二方面，在第二方面的某些实现方式中，所述第一类数据结构包括对应所述关键字的每个比特的测量信息。
例如，在本申请的实施例中，流对应的关键字为4比特，第一类数据结构可以包括对应该流关键字的4个比特的测量信息，即第一类数据结构可以包括对应于关键字的4个比特的测量信息。
结合第二方面，在第二方面的某些实现方式中，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
在本申请实施例的技术方案中，第一类数据结构包括的第一矩阵单元的数量与对应流的关键字的比特数相关，若对应流的关键字包括K个比特，则第一类数据结构可以包括1个至K个第一数据单元矩阵，其中，第一个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息，第K个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息；第二类数据单元结构包括一个第二数据单元矩阵，在第二数据单元矩阵中基于对应流的关键字记录测量信息，即将基于对应流的关键字的整体记录测量信息。
应理解，在本申请的实施例中，测量信息可以是报文数量，或报文长度，或与报文属性相关的参数。
结合第二方面，在第二方面的某些实现方式中，所述第二类数据结构中的M ij用于记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中 的报文数量或报文长度中的至少一项；
所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的所述第一测量信息，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述N为正整数。例如，第一测量信息和第二测量信息可以是统计值。
结合第二方面，在第二方面的某些实现方式中，所述第二类数据结构中的M ij用于记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，在一个示例中，所述第二类数据结构中的M ij用于记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为1时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，在一个示例中，所述第二类数据结构中的M ij用于记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为0时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
结合第二方面，在第二方面的某些实现方式中，所述第一流与M ij存在映射关系，所述映射关系为：h i(key)＝j；其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
在本申请的实施例的技术方案中，第一流与记录第一流统计值的数据单元M ij之间存在着映射关系，即h i(key)＝j，h i(x)是第i个hash函数。此外M ij可以记录多条流的统计值。这多条流有共同的特征，即通过相同的哈希函数根据哈希值索引到数据单元矩阵中相同的数据单元。
结合第二方面，在第二方面的某些实现方式中，所述控制面设备根据所述第一数据，得到统计结果，包括：所述控制面设备根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，所述第一集合包括从所述第一数据包括的r*c个栈stack中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一集合的残余流对应的所有的数据单元，所述r*c个栈中的任意一个栈为根据K+1个数据单元矩阵中的相同行和列的数据单元组成的数据单元组。
应理解，在K+1个数据单元矩阵中相同的行和列中可以共同记录了具备相同hash冲突的流，称这K+1个数据单元可以为一个栈(stack)，可以用(i,j)表示在K+1个数据单元矩阵中第i行j列的数据单元组成的数据单元组即一个栈stack，这样第一数据中可以包括 r*c个栈stack。
结合第二方面，在第二方面的某些实现方式中，所述控制面设备根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，包括：所述控制面设备从所述r*c个栈stack中的第一栈中析取大流，所述第一集合包括所述第一栈中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一栈中析取大流的残余流对应的所有的数据单元。
结合第二方面，在第二方面的某些实现方式中，所述方法还包括：若所述更新的所述第一数据满足正态分布，则确定所述第一集合包括所述第一数据中的所有大流。
在本申请的实施例中，根据定理即对于任何stack(i,j)及level k，如果stack(i,j)中不包括大流，则其概率分布服从正态分布。因此，基于定理可知若更新的第一数据中不包括大流，则其概率分布满足正态分布。
结合第二方面，在第二方面的某些实现方式中，所述方法还包括：若所述更新的所述第一数据不满足正态分布，则从所述第一数据包括的所述r*c个栈stack中析取大流。
在本申请的实施例中，根据定理即对于任何stack(i,j)及level k，如果stack(i,j)中不包括大流，则其概率分布服从正态分布。基于定理可知若更新的第一数据中的概率分布不满足正态分布，则说明更新的第一数据中还包括大流，因此需要从更新的第一数据中继续析取大流。
结合第二方面，在第二方面的某些实现方式中，所述方法还包括：若所述第一集合为空集，则调整阈值后从所述第一数据包括的所述r*c个栈stack中析取大流，其中，所述阈值表示大流流量与总流量的比值或大流频度与总流量频度的比值。
例如，阈值可以表示大流流量与总流量的最小比值，或者，可以表示大流频度与总流量频度的最小比值。
结合第二方面，在第二方面的某些实现方式中，所述调整阈值后从所述第一数据包括的所述r*c个栈stack中析取大流，包括：获取所述r*c个栈stack中对应一个比特的值为1的概率；根据所述概率确定第三集合，所述第三集合包括所述大流的候选关键字；根据所述第三集合确定所述大流的关键字。
结合第二方面，在第二方面的某些实现方式中，所述根据所述第三集合确定所述大流的关键字，包括：基于每比特的所述第一类数据结构与所述第二类数据结构中相同栈的比值确定K个频度，计算出所述第三集合包括的至少一个关键字的频度；根据每比特确定性概率对所述至少一个关键字进行筛选，确定所述大流的关键字；根据哈希函数对所述大流的关键字进行频度校验。
在本申请的实施例中，根据栈stack中对应一个比特的值为1的概率则可以确定候选关键字集合。例如，析取大流之前的栈(i,j)中第k个第一数据单元矩阵中记录的频度值与第二数据单元矩阵中的频度值的比值记为R i,j[k]。若R i,j[k]>θ，且1-R i,j[k]>θ(比如θ取0.99)，估计第k比特为1或0。如果不满足，我们给第k比特记作*，表示可以为1，也可以为0。
结合第二方面，在第二方面的某些实现方式中，所述统计结果包括所述第一数据中的大流列表，或所述第一数据中除大流外的残余流的数据，或所述第一数据中每比特的流数据的频度的分布中的至少一项。
结合第二方面，在第二方面的某些实现方式中，所述方法还包括：所述控制面设备接 收流量查询请求；所述控制面设备根据所述流量查询请求发送查询结果。
例如，控制面设备获取的查询请求可以是源于第三方应用的查询请求。
第三方面，本申请实施例提供了一种网络测量设备，所述网络测量设备适用于包括所述网络测量设备和控制面设备的系统中，所述网络测量设备包括：处理模块，用于测量第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应；通信模块，用于向所述控制面设备发送所述第一数据。
在本申请实施例的技术方案中，网络测量设备通过第一数据包括的对应流的关键字的比特级的第一类数据结构进行流数据的测量，将基于第一数据记录的测量信息上报至控制面设备。从而实现了通过一种通用的测量方法，可以支持大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量的应用。
结合第三方面，在第三方面的某些实现方式中，所述第一类数据结构包括对应流的关键字的至少一个比特的测量信息。
例如，在本申请的实施例中，假设流对应的关键字为4比特，第一类数据结构可以包括对应该流关键字的4个比特中的3比特或者2比特的测量信息，即第一类数据结构可以包括对应于关键字的部分比特的测量信息。
结合第三方面，在第三方面的某些实现方式中，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
结合第三方面，在第三方面的某些实现方式中，所述第一测量信息与所述流的关键字的每个比特对应。
结合第三方面，在第三方面的某些实现方式中，所述第一类数据结构包括对应所述关键字的每个比特的测量信息。
结合第三方面，在第三方面的某些实现方式中，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
在本申请实施例的技术方案中，第一类数据结构包括的第一矩阵单元的数量与对应流的关键字的比特数相关，若对应流的关键字包括K个比特，则第一类数据结构可以包括1个至K个第一数据单元矩阵，其中，第一个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息，第K个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息；第二类数据单元结构包括一个第二数据单元矩阵，在第二数据单元矩阵中基于对应流的关键字记录测量信息，即将基于对应流的关键字的整体记录测量信息。
应理解，在本申请的实施例中，测量信息可以是报文数量，或报文长度，或与报文属性相关的参数。
结合第三方面，在第三方面的某些实现方式中，所述处理模块，具体用于在所述第二数据单元矩阵中的M ij记录第一流的所述第一测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第一测 量信息包括所述第一流包括的报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为预设值时，在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的所述第二测量信息，所述N小于或等于所述K，所述N为正整数。
例如，第一测量信息和第二测量信息可以是统计值。
结合第三方面，在第三方面的某些实现方式中，所述预设值为1。
例如，在一个示例中，所述网络测量设备在所述第二数据单元矩阵中的M ij记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为1时，所述测量设备在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
结合第三方面，在第三方面的某些实现方式中，所述预设值为0。
例如，在一个示例中，所述网络测量设备在所述第二数据单元矩阵中的M ij记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为0时，所述测量设备在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
结合第三方面，在第三方面的某些实现方式中，所述第一流与M ij存在映射关系，所述映射关系为：h i(key)＝j；其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
第四方面，本申请实施例提供了一种控制面设备，所述控制面设备适用于包括网络测量设备和所述控制面设备的系统中，所述控制面设备包括：通信模块，用于获取所述网络测量设备上报的第一数据，所述第一数据包括第一类数，所述第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应；处理模块，根据所述第一数据，得到统计结果。
在本申请的实施例中，控制面设备接收网络测量设备上报的测量信息，网络测量设备通过第一数据包括的对应流的关键字的比特级的第一类数据结构进行流数据的测量，将基于第一数据记录的测量信息上报至控制面设备，控制面设备根据第一数据的概率分布特性，得到统计结果。从而实现了通过一种通用的测量方法，可以支持大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量的应用。
结合第四方面，在第四方面的某些实现方式中，所述第一类数据结构包括对应流的关键字的至少一个比特的测量信息。
例如，在本申请的实施例中，假设流对应的关键字为4比特，第一类数据结构可以包括对应该流关键字的4个比特中的3比特或者2比特的测量信息，即第一类数据结构可以包括对应于关键字的部分比特的测量信息。
结合第四方面，在第四方面的某些实现方式中，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对 应。
结合第四方面，在第四方面的某些实现方式中，所述第一测量信息与所述流的关键字的每个比特对应。
结合第四方面，在第四方面的某些实现方式中，所述第一类数据结构包括对应所述关键字的每个比特的测量信息。
例如，假设流对应的关键字为5比特，第一类数据结构可以包括对应该流关键字的5个比特的测量信息，即第一类数据结构可以包括对应于关键字的5个比特中的每个比特的测量信息。
结合第四方面，在第四方面的某些实现方式中，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
在本申请实施例的技术方案中，第一类数据结构包括的第一矩阵单元的数量与对应流的关键字的比特数相关，若对应流的关键字包括K个比特，则第一类数据结构可以包括1个至K个第一数据单元矩阵，其中，第一个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息，第K个第一数据单元矩阵与对应流的关键字的第一比特相对应记录测量信息；第二类数据单元结构包括一个第二数据单元矩阵，在第二数据单元矩阵中基于对应流的关键字记录测量信息，即将基于对应流的关键字的整体记录测量信息。
应理解，在本申请的实施例中，测量信息可以是报文数量，或报文长度，或与报文属性相关的参数。
结合第四方面，在第四方面的某些实现方式中，所述第二类数据结构中的M ij用于记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的所述第一测量信息，述第一测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述N为正整数。
结合第四方面，在第四方面的某些实现方式中，所述预设值为1。
例如，在一个示例中，所述第二类数据结构中的M ij用于记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为1时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
结合第四方面，在第四方面的某些实现方式中，所述预设值为0。
例如，在一个示例中，所述第二类数据结构中的M ij用于记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第 N比特的取值为0时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。结合第四方面，在第四方面的某些实现方式中，所述第一流与M ij存在映射关系，所述映射关系为：h i(key)＝j；其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
在本申请的实施例的技术方案中，第一流与记录第一流统计值的数据单元M ij之间存在着映射关系，即h i(key)＝j，h i(x)是第i个hash函数。此外M ij可以记录多条流的统计值。这多条流有共同的特征，即通过相同的哈希函数根据哈希值索引到数据单元矩阵中相同的数据单元。
结合第四方面，在第四方面的某些实现方式中，所述处理模块，具体用于根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，所述第一集合包括从所述第一数据包括的r*c个栈stack中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一集合的残余流对应的所有的数据单元，所述r*c个栈中的任意一个栈为根据K+1个数据单元矩阵中的相同行和列的数据单元组成的数据单元组。
应理解，在K+1个数据单元矩阵中相同的行和列中可以共同记录了具备相同hash冲突的流，称这K+1个数据单元可以为一个栈(stack)，可以用(i,j)表示在K+1个数据单元矩阵中第i行j列的数据单元组成的数据单元组即一个栈stack，这样第一数据中可以包括r*c个栈stack。
结合第四方面，在第四方面的某些实现方式中，所述处理模块，具体用于从所述r*c个栈stack中的第一栈中析取大流，所述第一集合包括所述第一栈中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一栈中析取大流的残余流对应的所有的数据单元。
结合第四方面，在第四方面的某些实现方式中，若所述更新的所述第一数据满足正态分布，所述处理模块，还用于确定所述第一集合包括所述第一数据中的所有大流。
在本申请的实施例中，根据定理即对于任何stack(i,j)及level k，如果stack(i,j)中不包括大流，则其概率分布服从正态分布。基于定理可知若更新的第一数据中的概率分布不满足正态分布，则说明更新的第一数据中还包括大流，因此需要从更新的第一数据中继续析取大流。
结合第四方面，在第四方面的某些实现方式中，若所述更新的所述第一数据不满足正态分布，所述处理模块，还用于从所述第一数据包括的所述r*c个栈stack中析取大流。
结合第四方面，在第四方面的某些实现方式中，若所述第一集合为空集，所述处理模块，具体用于调整阈值后从所述第一数据包括的所述r*c个栈stack中析取大流，其中，所述阈值表示大流流量与总流量的比值或大流频度与总流量频度的比值。
例如，阈值可以表示大流流量与总流量的最小比值，或者，可以表示大流频度与总流量频度的最小比值。
结合第四方面，在第四方面的某些实现方式中，所述处理模块，具体用于获取所述r*c个栈stack中对应每比特的值为1的概率；根据所述概率确定第三集合，所述第三集合包括所述大流的候选关键字；根据所述第三集合确定所述大流的关键字。
结合第四方面，在第四方面的某些实现方式中，所述处理模块，具体用于基于每比特的所述第一类数据结构与所述第二类数据结构中相同栈的比值确定K个频度，计算出所述 第三集合包括的至少一个关键字的频度；根据每比特确定性概率对所述至少一个关键字进行筛选，确定所述大流的关键字；根据哈希函数对所述大流的关键字进行频度校验。
在本申请的实施例中，根据栈stack中对应每比特的值为1的概率则可以确定候选关键字集合。例如，析取大流之前的栈(i,j)中第k个第一数据单元矩阵中记录的频度值与第二数据单元矩阵中的频度值的比值记为R i,j[k]。若R i,j[k]>θ，且1-R i,j[k]>θ(比如θ取0.99)，估计第k比特为1或0。如果不满足，我们给第k比特记作*，表示可以为1，也可以为0。
结合第四方面，在第四方面的某些实现方式中，所述统计结果包括所述第一数据中的大流列表，或所述第一数据中除大流外的残余流的数据，或所述第一数据中每比特的流数据的频度的分布中的至少一项。
结合第四方面，在第四方面的某些实现方式中，所述通信模块，还用于所述控制面设备接收流量查询请求；所述处理模块，还用于根据所述流量查询请求发送查询结果。
例如，控制面设备获取的查询请求可以是源于第三方应用的查询请求。
第五方面，提供了一种网络测量设备，该网络测量设备包括：存储器，用于存储计算机程序；处理器，用于执行所述存储器中存储的计算机程序，以使得所述装置执行上述第一方面或第一方面中的任一种可能实现方式中的方法。
第六方面，本申请实施例提供一种控制面设备，该控制面设备包括：存储器，用于存储计算机程序；处理器，用于执行所述存储器中存储的计算机程序，以使得所述装置执行上述第二方面或第二方面中的任一种可能实现方式中的方法。
第七方面，提供了一种系统，该系统包括至少一个上述任一方面或任一方面中的任一种可能实现方式中的网络测量设备和上述任一方面或任一方面中的任一种可能实现方式中的控制面设备。
第八方面，本申请实施例提供一种计算机可读存储介质，该计算机可读存储介质存储用于实现第一方面或第一方面的任一种可能的实现方式所述的方法的指令。
第九方面，本申请实施例提供一种计算机可读存储介质，该计算机可读存储介质存储用于实现第二方面或第二方面的任一种可能的实现方式所述的方法的指令。
第十方面，本申请提供了一种包含指令的计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述第一方面或第一方面的任一种可能的实现方式所述的方法。
第十一方面，本申请提供了一种包含指令的计算机程序产品，当该计算机程序产品在计算机上运行时，使得计算机执行上述第二方面或第二方面的任一种可能的实现方式所述的方法。
附图说明
图1示出了基于概要(sketch based)的测量技术的数据结构的示意图。
图2示出了现有技术中在网络设备中记录流的方法的示意图。
图3示出了现有技术中在中心控制器中恢复流的方法的示意图。
图4是根据本申请实施例应用的场景的一种实现方式的示意图。
图5是根据本申请一个实施例的网络流测量的方法的交互流程图。
图6是根据本申请的网络流测量的方法的数据结构的示意图。
图7是根据本申请一个实施例的控制面设备的执行网络流量测量方法的示意图。
图8是根据本申请一个实施例的控制面设备从栈stack中析取大流的算法示意图。
图9是根据本申请一个实施例的网络测量设备的示意性框图。
图10是根据本申请一个实施例的控制面设备的示意性框图。
图11是根据本申请另一个实施例的网络测量设备的示意性框图。
图12是根据本申请另一个实施例的控制面设备的示意性框图。
具体实施方式
下面将结合附图，对本申请中的技术方案进行描述。
为了更好地理解本申请实施例的网络流测量方法，下面先对相关的一些基本概念进行简单说明。
概要技术(count-min sketch，CMS)是一个次线性空间数据结构。如图1所示，图1示出了CMS结构的示意图。CMS是由二维数组构成，数组的每个元素表示一个计数值，即计数值[1,1],…,计数值[d,w]。数组的每个元素初始化为0，d个相互独立的Hash函数被均匀、随机地选择。如图1所示CMS采用固定d个哈希表，每个哈希表宽度为w，对于获取的每一个报文，基于报文的关键字进行多次哈希操作。在每一个哈希表中，以哈希值为索引到特定哈希桶位置，累计记录报文的大小。
例如，若需要查询特定流f的大小，采用相同的几个哈希函数定位到多个桶，从中确定取值最小的桶，其值为对流f大小的估计值。理论可以证明，选取特定的w,d可以达到相当高的多流查询精度。
流数据：通常是指五元组(包括源IP、目的IP、协议号、源端口和目的端口)相同的报文流。
大流：某流的流量超过总流量百分比阈值的流。
流突变：在一定的时间间隔内，某流的流量变化超过一定百分比阈值的流
流熵：基于流的频度分布的计算。
在现有技术中，能够基于以下的FlowRadar方法检测链路上所有流，并记录流大小。即在网络设备中部署传感器执行以下算法：
如图2所示，对于每个流报文的关键字(key)，首先可以利用K f个哈希函数在Bloom Filter(Flow Filter表)中测量是新数据流还是旧数据流。
若在Flow Filter表中的计数值全是1，则表示测量的为旧数据流；若在Flow Filter表中的计数值出现了0，则表示测量的为新数据流，将对应位置的计数值从0置为1。
在流计数值(Flow Counter)表中，采用K C个哈希函数指向索引记录。
对于新数据流：FlowCnt+1&&PktCnt+＝报文长度；
对于旧新数据流：PktCnt+＝报文长度。
其中，FlowCnt可以表示报文数量的计数值，PktCnt可以表示报文的长度或者表示报文的数量。例如，图2中所示，对于报文a，其FlowCnt值为1，PktCnt值为size(a)。
基于上述记录的Flow Counter表，在网络设备中部署的传感器周期性的将记录的Flow Counter表送至中心控制器。在中心控制器，FlowRadar进行解码算法对所有的流进行恢复。
在中心控制器中，FlowRadar恢复的方法。如图3所示，图3为中心控制器恢复流的 方法的示意图。如图3所示，中心控制器在恢复流时，首先，从Flow Counter表中，先找到Flow Cnt(流的计数值)＝1的项；接着输出该项的FlowXor(流的异或)作为flow的key，PacketCnt作为flow的大小；然后对key做K c个hash，将其余位置对应Cnt及PktCnt减掉。基于这样“剥洋葱”式可以完成所有流记录恢复。
但是，当中心控制器FlowRadar解码时找不到FlowCnt值为1的项，则无法基于上述算法进行流的恢复，从而导致解码失败。此外，FlowRadar所实现的网络测量功能，主要针对于流的信息，基于此可以实现流数量，流突变，流分布等查询。但是对于大流检测等常用检测功能，它并不能很好支持。因此，上述的流检测的方法的通用性较差。
随着大数据、人工智能技术的发展推动网络智能化，实现网络智能化的基础是网络全网全量的数据测量。而对网络测量技术，有两个关键的要求。
要求1：高性能，即需要做到基于每个报文进行信息提取和统计。
随着网络链路速率的不断提高和网络数据流的急剧增加，当前在高速网络链路上，网络流量测量需要极高的计算和存储资源。在高速链路上，处理每个分组必须按秒级时间，从而给网络流量测量技术带来了很大挑战。
要求2：通用性，即对于网络流有多方面的测量数据要求。
例如，需要实时监视链路上每一条流，需要掌握网络链路上的大流，流的大小分布，检测是否有突变的流量，检测网络中是否有各种网络攻击事件等。希望采用一两个少量的通用传感器以支撑众多的测量目标。
根据图2和图3所示的网络测量的方法，针对需求2满足度较差。许多简单的基于概要(sketch-based)的算法支持的查询是有限。
例如，上文举例的CMS，可以很容易查询“流f的大小是否超过阈值T”的查询，但是对于查询“找出所有超过阈值T的流”，就需要枚举所有可能的流。而枚举海量的流在计算上是不可接受的。
此外，sketch-based测量技术针对不同的待测量信息，通常需要不同的测量算法。当一个实际的测量任务需要多种信息时，就要同时部署多个sketch-based算法。多个sketch-based算法的同时运行，将带来更多的处理与存贮资源开销。
有鉴于此，本申请提出了一种网络测量的方法和装置，通过一种通用的测量方法，利用系统中的一种算法，可以实现大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量应用。
下面将结合具体的例子详细描述本申请实施例。应注意，这只是为了帮助本领域技术人员更好地理解本申请实施例，而非限制本申请实施例的范围。
图4是根据本申请的实施例的应用场景的示意图。
如图4所示，系统中可以包括控制面设备101和多个节点(例如网络测量设备)102。控制面设备101用于接收系统中的多个节点102发送的测量数据，例如接收多个节点102发送的第一数据。
节点102可以为具有处理功能的设备，例如可以包括当前技术已知的任何计算设备，如服务器、台式计算机等等。
节点102中可以包括存储器和处理器。存储器可以用于存储程序代码，例如，操作系统以及其他应用程序。处理器可以用于调用存储器存储的程序代码，以实现节点的相应功 能。节点中包括的处理器和存储器可以通过芯片实现，此处不作具体的限定。
例如，图4所示的系统可以分布式的数据平面和集中式的控制平面。可以将控制面设备101看作是集中式的控制平面。将多个节点102看作是分布式的数据平面。
节点处可以安装有操作系统以及其他应用程序。例如，节点102上都可以部署一个采集模块。采集模块运行测量算法，当数据报文进入设备时，采集模块从转发模块获取报文信息，执行测量算法更新计算，结果保存在数据结构中。例如，测量算法可以是多概要算法(multi-sketch)。
节点102可以周期性的向控制面设备101上报测量数据。例如，采集模块定期将该数据结构上报给控制面。
在集中式的控制平面，可以包括数据整合模块用于收集所有设备上的数据结构，进行处理得到分类的数据流。
例如，可以得到大流及流关键字比特级概率分布计算，输出得到大流表，小流残余及比特级流分布数据。基于这些数据，支持众多流统计查询，如大流查询，流大小查询，流突变查询，流数量查询，流分布查询等。
应理解，在本申请的各实施例中，“第一”、“第二”等仅是为了指代不同的对象，并不表示对指代的对象有其它限定。
下面将结合图5，具体介绍本申请的实施例中在包括网络测量设备和控制面板设备的系统中的网络流测量的过程。
图5是根据本申请一个实施例的网络流测量的方法的流程示意图。其中，网络测量设备可以是系统中多个测量设备中的任意一个测量设备，例如，可以是图4中的任意一个节点。
在本申请的实施例中，系统中可以包括多个网络测量设备和一个控制面设备，控制面设备对多个网络测量设备提供的测量数据进行处理，从而可以得到分类的流数据。
S210、网络测量设备测量第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括第一测量信息，所述第一测量信息与所述流的关键字的比特对应。
可选地，第一类数据结构可以包括对应流的关键字的至少一个比特的第一测量信息。
例如，第一类数据结构可以包括对应流的关键字的部分比特的第一测量信息，或者，第一类数据结构也可以包括对应流的关键字的每个比特的第一测量信息。
需要说明的是，第一数据是基于关键字的比特信息构建的数据结构。
例如，对应流的关键字为3比特，则在一个示例中第一数据中包括的第一类数据结构可以包括对应流的关键字的第1比特的第一测量信息；在一个示例中第一数据中包括的第一类数据结构可以包括对应流的关键字的3个比特中的每个比特的第一测量信息。本申请实施例对此不作限定。
可选地，第一数据还可以包括第二类数据结构，第二类数据结构包括对流的关键字的测量信息，即第二类数据结构中记录的测量信息基于流的关键字。
可选地，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
应理解，第一测量信息与第二测量信息可以包括流的报文长度、流的数量以及与流属性相关的参数。其中，“第一”、“第二”仅指代不同，本申请对此不作限定。
应理解，第一类数据结构中是基于对应流关键字的比特进行第一测量信息的记录，第二类数据结构是基于对应流关键字整体进行第二测量信息的记录。
例如，图6中以第一数据中包括第一类数据结构和第二类数据结构，其中，第一类数据结构可以包括对应所述关键字的每个比特的第一测量信息，第二类数据结构可以包括对应所述流的关键字的第二测量信息进行举例说明。
在本申请的实施例中，图6所示的数据结构可以是多概要算法(multi-sketch)。如图6所示，假设流ID(例如，流关键字flowkey)为L位比特，则构建L+1个概要(sketch)数据结构。每个sketch数据结构对应每个层(level)，每个sketch数据结构可以是一个具有r行，c列的数据单元矩阵，其中，一个小格表示一个数据单元。
每一个sketch对报文的关键字，进行r次hash，每个hash函数独立均匀地将关键字映射到[1..c]空间。其中，定义V i,j[k]表示第k个sketch中数据单元矩阵中的第i行，第j列的值，则multi-sketch为如图6示的流程。
可选地，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
可选地，所述第一类数据结构为K个第一数据单元矩阵，其中，一个第一数据单元矩阵用于记录所述对应流的关键字的一比特的测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述流的关键字的测量信息。
例如，第一类数据结构可以看作是level 1至level L的数据结构，用于记录所述对应流的关键字的一比特的第一测量信息；第二类数据结构可以看作是leve 0的数据结构，用于记录所有流的关键字的第二测量信息。即第一类数据结构为K个第一数据单元矩阵，所述第二类数据结构为第二数据单元矩阵。其中，K个第一数据单元矩阵也就是L个数据结构。
例如，第一测量信息和第二测量信息可以是统计值。
可选地，网络测量设备在第二数据单元矩阵中的M ij记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为预设值时，所述测量设备在所述第二类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，在一个示例中，在所述第一流关键字的第N比特的取值为1时，所述测量设备在所述第二类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，在一个示例中，在所述第一流关键字的第N比特的取值为0时，所述测量设备在所述第二类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
应理解，第一流与M ij之间存在映射关系，通过将第一流的关键字通过哈希函数的计算，以哈希值为索引至数据结构中的单元，即h i(key)＝j，其中，h i()表示第i个哈希函数，key表示第一流对应的关键字。
例如，在sketch 0中可以记录所有关键字的统计值。例如，记录所有关键字的数量或者长度。
sketch-1至sketch-L则将关键字中对应每个比特位的值为预设值的进行统计。例如，预设值可以是1或者预设值也可以是0，本申请实施例对此不作限定。
例如，以sketch-1至sketch-L则将关键字中对应每个比特位的值为‘1’的进行统计为例进行说明。
例如，记f[k]表示流ID第k位bit的值，则当第k位为1时，f[k]＝1,；当第k位不为1时，f[k]＝0。
对level k的数据单元记录如下：
对k属于[1...L]，若f[k]＝1，则V i,j[k]的值增加1；
否则，将不对V i,j[k]的值进行操作。
需要说明的是，若统计流报文个数则增加1,如果统计流字节单位的大小，则值增加报文按字节记录的长度值，以下如没有特别说明，按流报文个数说明，不再重复说明。
具体地，图6的示例中是以L＝4两个分别属于流ID关键字1＝(0101)，关键字2＝(1011)，r＝2的情况进行了举例说明记录数据单元的示意图。
在一个示例中，以统计流报文的个数举例说明。
例如，图6所示的关键字1为0101，通过h 1哈希函数将哈希函数索引到level0数据单元的第1行第2列；通过h2哈希函数将哈希函数索引到level0数据单元的第2行第7列。逐一比特位的对关键字进行统计，例如，关键字1的第一比特位的值为不是1，则不对V 1,2[1]的值及V 2,7[1]的值进行操作，即不对level 1中第1行第2行的值进行操作；关键字1的第二比特位的值为1，则V 1,2[2]的值与V 2,7[2]的值增加1，即对level 2中第1行第2行的值进行操作；关键字1的第三比特位的值不是1，则不对V 1,2[3]的值及V 2,7[3]的值进行操作，即不对level 3中第1行第2行的值进行操作；关键字1的第四比特位的值为1，则V 1,2[4]的值与V 2,7[4]的值增加1，即对level 4中第1行第2行的值进行操作。
在一个示例中，以统计流报文的长度值举例说明。
例如，图6所示的关键字1为0101，通过h1哈希函数将哈希函数索引到level0数据单元的第1行第2列；通过h2哈希函数将哈希函数索引到level0数据单元的第2行第7列。逐一比特位的对关键字进行统计，例如，关键字1的第一比特位的值为不是1，则不对V 1,2[1]的值及V 2,7[1]的值进行操作，即不对level 1中第1行第2行的值进行操作；关键字1的第二比特位的值为1，则V 1,2[2]的值与V 2,7[2]的值增加关键字1对应的报文长度值，即对level 2中第1行第2行的值进行操作；关键字1的第三比特位的值不是1，则不对V 1,2[3]的值及V 2,7[3]的值进行操作，即不对level 3中第1行第2行的值进行操作；关键字1的第四比特位的值为1，则V 1,2[4]的值与V 2,7[4]的值增加关键字1对应的报文长度值的，即对level 4中第1行第2行的值进行操作。
上述以关键字1举例说明，对于关键字2与上述过程类似，此处不再赘述。
在一个示例中，以sketch-1至sketch-L则将关键字中对应每个比特位的值为‘0’的进行统计为例进行说明。
例如，记f[k]表示流ID第k位bit的值，则当第k位为0时，f[k]＝1,；当第k位不为0时，f[k]＝0。
对level k的数据单元记录如下：
对k属于[1...L]，若f[k]＝0，则V i,j[k]的值增加1；
否则，将不对V i,j[k]的值进行操作。
需要说明的是，若统计流报文个数则增加1,如果统计流字节单位的大小，则值增加报文按字节记录的长度值，以下如没有特别说明，按流报文个数说明，不再重复说明。
具体统计过程与上述对应每个比特位的值为‘1’的进行统计类似，此处不再赘述。
图6示出了网络测量设备测量第一数据，将统计结果记录数据单元的过程。在系统中，可以存在多个网络测量设备，每个测量设备基于第一数据的结构将测量信息统计，周期性的发送至系统中的控制面设备。
S220、网络测量设备向所述控制面设备发送所述第一数据。
在本申请的实施例中，控制面设备接收网络测量设备发送的第一数据，根据所述第一数据，得到统计结果。
可选地，统计结果可以包括第一数据中的大流列表，或所述第一数据中除大流外的残余流的数据，或所述第一数据中每比特的流数据的频度的分布中的至少一项。
应理解，基于统计结果，可以支持众多流统计查询。
例如，支持大流查询，流大小查询，流突变查询，流数量查询，流分布查询等。
在本申请的实施例中，第一数据可以包括第一类数据结构和第二类数据结构，所述第一类数据结构可以包括对应所述关键字的比特的测量信息。例如，第一类数据结构可以包括对应关键字的至少一个比特的测量信息，或者，第一类数据结构可以包括对应关键字的每个比特的测量信息。第一数据中还可以包括第二类数据结构，所述第二类数据结构包括对应流的关键字的测量信息。
可选地，所述第二类数据结构中的M ij用于记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；
所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，所述第一类数据结构为K个第二数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为1时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，所述第一类数据结构为K个第二数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为0时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
应理解，第一流与M ij之间存在映射关系，通过将第一流的关键字通过哈希函数的计算，以哈希值为索引至数据结构中的单元，即h i(key)＝j，其中，h i()表示第i个哈希函数，key表示第一流对应的关键字。
例如，如图6中所示，第二类数据结构可以为level 0的数据结构。假设第一流关键字为图6所示的关键字1，则第一类数据结构可以为level 1至level 4的数据单元。M ij可以表示为每个数据结构中的第1行第2列或者第2行第7列，即对应于在level 0数据单元中 记录关键字1的位置的行和列的信息。
应理解，上述仅为以第一流关键字为关键字1进行举例说明，并不对本申请的实施例作任何限定。
下面将简要说明控制面设备执行的算法的相关理论。
应理解，第一数据包括K+1个单元矩阵，即包括第二数据单元矩阵和K个第一数据单元矩阵，其中，第一数据单元矩阵的个数可以与流关键字的比特数相关。
例如，若第一类数据结构包括对应所述关键字的每个比特的测量信息。假设流关键字为3比特，则第一数据中可以包括3个第一数据单元矩阵。
在K+1个数据单元矩阵中，相同的行和列中记录了具备相同hash冲突的流，称这K+1个数据单元组为一个栈(stack)，用(i,j)表示第i行j列的stack，这样共有r*c个stack栈。用n i,j表示哈希到(i,j)的流数量，用n表示观察到的所有流数量，则有
Figure PCTCN2019097092-appb-000001
用p[k]表示第k个比特位的值是‘1’的概率。
有如下两个基本假设：
假设1：哈希函数的均匀性。每一个hash函数将大量的流均匀地分散在多列中。
Figure PCTCN2019097092-appb-000002
假设2：所有哈希函数h i(f)与p[k]是独立的，即对于所有
p i,j[k]＝p[k](1≤i≤r,1≤j≤c)
存在如下定理：
定理1：对于任何stack(i,j)及level k，如果stack(i,j)中不包括大流，则
R i,j[k]＝V i,j[k]/V i,j[0]
服从高斯分布N(p[k],σ 2[k])，其中，σ 2[k]＝p[k](1-p[k])c/n。
定理1：要求在一个stack中没有大流，本申请实施例的基本思想是如果从multi-level中析出所有大流，则剩下的流构建的sketch结果，就服从高斯分布。
利用比特级的sketch与sketch0的关系，可以逐比特对大流进行析出，并通过校验剩余的sketch是否符合高斯分布，可以判定迭代是否完成。而这个基本思想，可以利用假设2，在每个stack上进行操作，实现所有大流析出。
基于解析出的大流及残余sketch数据，可以实现流列表，流大小，流突变，流数量，流分布等多种网络流统计查询。
可选地，所述控制面设备根据所述第一数据，得到统计结果，包括：
所述控制面设备根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，所述第一集合包括从所述第一数据包括的r*c个栈stack中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一集合的残余流对应的所有的数据单元，所述r*c个栈中的任意一个栈为根据K+1个数据单元矩阵中的相同行和列的数据单元组成的数据单元组。
也就是说，控制面设备根据关键字的每比特上的分布特性(例如概率分布)获取第一集合和更新的所述第一数据，控制面设备在第一数据包括的所有stack栈即r*c个栈stack 中进行析取大流的操作，将析取的所有栈的大流记录在第一集合中，则更新的第一数据包括所述r*c个栈stack中除所述第一集合的残余流对应的所有的数据单元。
应理解，在控制面设备在第一数据包括的所有stack栈即r*c个栈stack中进行析取大流的操作中，控制面设备可以是基于所有栈进行一次析取大流后更新一次第一数据。
例如，控制面设备从第一栈中析取大流，第一栈是r*c个栈stack中的任意一个，第一集合可以包括第一栈中析取的大流，更新的第一数据可以包括r*c个栈stack中除第一栈中析取大流的残余流对应的所有的数据单元。
也就是说，第一集合可以包括第一数据中的大流，更新的第一数据可以包括第一数据中除大流外的残余流。
可选地，若所述更新的所述第一数据满足正态分布，则确定所述第一集合为大流。
根据上述定理1可知，若一个栈中不包括大流，则其概率分布特性服从高斯分布。即若更新的第一数据满足正态分布，则说明更新的第一数据中不包括大流，也就意味着第一集合中包括的析取的大流为大流。
可选地，若所述更新的所述第一数据不满足正态分布，则从所述第一数据包括的所述r*c个栈stack中析取大流。
需要说明的是，若更新的第一数据不满足正态分布，则说明在更新的第一数据中可能还包括未析取的大流，则继续从第一数据中的所有栈中析取大流。析取出的大流记录在第一集合中，当更新的所述第一数据满足正态分布，则可以说明第一数据中的大流已经全部析取，则此时第一集合为大流。
可选地，若更新的第一数据不满足正态分布且第一集合为空集，则调整阈值后从所述第一数据包括的所述r*c个栈stack中析取大流，其中，所述阈值表示大流流量与总流量的比值或大流频度与总流量频度的比值。
需要说明的是，阈值可以是θ，θ表示一个判断大流的参数，若一个流的频度或大小f>θN，则称这个流是大流，其中N是一段观测时间内所有的流频度或大小。
根据上述定理1可知，若任意一个栈中不包括大流，则其概率分布特性服从高斯分布。当更新的第一数据不满足正态分布时，则说明更新的第一数据中还包括大流。
在一个实例中，从第一数据包括的所有栈，即从第一数据包括的r*c个栈stack中析取大流可以包括以下流程，控制面设备获取r*c个栈stack中对应每比特的值为1的概率；根据所述概率确定第三集合，所述第三集合包括所述大流的候选关键字；根据所述第三集合确定所述大流的关键字。
可选地，根据所述第三集合确定所述大流的关键字，包括：
基于每比特的所述第一类数据结构与所述第二类数据结构中相同栈的比值确定K个频度，计算出所述第三集合包括的至少一个关键字的频度；
根据每比特确定性概率对所述至少一个关键字进行筛选，确定所述大流的关键字；
根据哈希函数对所述大流的关键字进行频度校验。
需要说明，关于如何从栈中析取大流的具体过程可以参见后续图8中的描述。
在本申请的实施例中，通过第一数据包括的对应流的关键字的比特级的第一类数据结构，实现了通过一种通用的测量方法，可以支持大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量的应用。
图7是根据本申请一个实施例的控制面板设备的执行网络流量测量方法的示意图。
图7中网络流量测量方法可以是系统中的控制面设备执行的。例如，可以是图4中的控制面设备101。该方法包括：
S310、计算每个比特-层的分布概率。
例如，控制面设备在每个sketch(1到L)中计算均值与方差。
S320、基于每个栈(i,j)析取大流到集合F’。
S330、记录大流集合F，并从现有L+1个层的数据结构中删除集合F’。
S340、重新计算每个比特-层的分布特性N。
例如，基于删除大流，对stack(i,j)也进行删除工作，然后重新计算比特-层(bit-level)的分布。
S350、判断删除大流后的残余流是否满足正态分布。若满足正态分布，则执行S370输出大流集合F，残余流的数据结构以及基于bit-level的分布N。若不满足正态分布，则执行S360。
S360、判断F’是否为空集。若F’是空集，则调整阈值θ。
例如，如果不满足正态分布，则判断F’集合是否是空集。如果是空集，则调整域值θ，继续执行S320。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
图7描述了上述步骤的流程图，下面将说明关于执行图7所示算法的伪码实现。
Algorithm 1 Model Inference(算法1模型推理)
Input:Multi-level sketch S＝{V i,j[k]|0≤k≤L，1≤i≤r，1≤j≤c}
1：Large flow list
Figure PCTCN2019097092-appb-000003
2：θ＝1/2；
3：Bit-level counter distributions{N(p[k]，σ 2[k])}＝Computers Dist(S)
4：while true do
5：set of extracted large flows
Figure PCTCN2019097092-appb-000004
6：for all stack(i,j),1≤i≤r，1≤j≤c do；
7：F’＝F’∪ExtractLagerFlows(θ,(i,j),S,{N(p[k]，σ 2[k])})；
8：F＝F∪F’；
9：RemoveFlows(S,F’)；
10：{N(p[k]，σ 2[k])}＝Computers Dist(S)；
11：if Terminate({N(p[k]，σ 2[k])})then；
12：break；
13：if
Figure PCTCN2019097092-appb-000005
then；
14：θ＝θ/2.
上述代码为可以执行图7所示的流程图的伪码。
关于上述图7中步骤S330基于每个栈(i,j)析取大流到集合F’的流程，如图8所示。图8是根据本申请一实施例的从栈中析取大流的算法示意图。根据图8所示的流程，控制 面设备可以从第一数据包括的r*c个栈中析取大流至第一集合。
S410、利用数据结构中的数据，采用最大似然估计法，获得栈(i,j)中对应每bit值为1的概率
Figure PCTCN2019097092-appb-000006
例如，假设
R i,j[k]＝V i,j[k]/V i,j[0]
其中，k属于[1…L]，R i,j[k]是析取大流之前的栈(i,j)中第k个level的sketch中记录的频度值与level-0中的频度值的比值。
若R i,j[k]<θ，则对应第k个比特中没有大流记录，记
Figure PCTCN2019097092-appb-000007
若1-R i,j[k]<θ，则估计
Figure PCTCN2019097092-appb-000008
如果上述两种条件都不满足，即R i,j[k]>θ且1-R i,j[k]>θ，假设存在大流第k比特为1,并将此大流的记录从sketch[k]中移除后，观察余下的R i,j[k]与p[k]近似关系，利用贝叶斯定理可以计算出
Figure PCTCN2019097092-appb-000009
S420、利用每比特概率构建候选大流的流关键字，用哈希函数校验正确性。
例如，根据
Figure PCTCN2019097092-appb-000010
或
Figure PCTCN2019097092-appb-000011
(比如θ取0.99)估计第k比特的值为1或0。如果均不满足，将第k比特记作*，表示第k比特的值可以为1也可以为0。
根据得到的
Figure PCTCN2019097092-appb-000012
构建所有可能的flowkey，并将flowkey用原先的hash函数校验，确保h i(flowkey)＝j其中(i,j)即栈(i,j)。
S430、利用大流对均值的影响关系，估测对应流关键字的频度。
例如，从sketch S中去删除一个大流f后，假设大流频度为S f，对均值p[k]的影响可用下式表示：
若第k比特的值为1，则
Figure PCTCN2019097092-appb-000013
若第k比特的值为0，则
Figure PCTCN2019097092-appb-000014
根据上述等式可以计算出基于每比特的大流频度S f；
若第k比特的值为1，则
Figure PCTCN2019097092-appb-000015
若第k比特的值为0，则
Figure PCTCN2019097092-appb-000016
根据上述每比特的大流频度S f，取每个比特的频度均值即得到大流f的频度。
S440、记
Figure PCTCN2019097092-appb-000017
为构建候选选大流的流关键字的比特-层的确定性概率，并进行过滤。
例如，若f[k]＝1，则记
Figure PCTCN2019097092-appb-000018
为确定性概率；
若f[k]＝0，则记
Figure PCTCN2019097092-appb-000019
为确定性概率。这样，如果一个flowkey的大多数比特对应的确定性概率都接近1，则这个flowkey就很可能是要找的一个大流的flowkey。
例如，取50％比特数量的确定性概率大于90％的flowkey通过过滤，其它的进行丢弃。
S450、用其它哈希函数对流关键字进行频度校验。
例如，每个sketch有r个hash表，对于从(i,j)得到的大流集合C，利用其它hash 校验是否也都满足大流条件。假如其它hash到栈(s,t)s≠i，则如果出现任何一个k，k∈[1..L]，V s,t[k]<θ则V i,j[0]则校验不通过，将对应的流从大流集合C中删除。
基于图7和图8所示的流程，控制面设备可以得分类的流数据。
例如，可以得到大流表，记录大流的flowkey及对应的频度；残余流的数据结构记录，类似于一个如图6描述的multi-level sketch的结构，只是其中去除了大流的影响；比特级R i,j[k]的分布。
根据上述三种数据可以支持可以灵活支持多种流的查询。
例如，大流查询。基于大流的阈值T h，在大流表中可以直接判断输出。
例如，任意流频度查询。可以首先在大流表中查询，如果查不到表明是小流，在residual sketch中利用正向hash查询。
例如，流突变查询，记录两个周期的数据，在大流表中查询，同时在另一个周期数据中进行per-flow-frequency查询。
例如，流分布查询，利用residual sketch数据，采用MRAC算法得到流分布，进一步可用流分布数据计算流熵entropy。
例如，根据以下等式进行流数量的查询：
基于定理1，根据等式n＝(p[k](1-p[k])c/(σ 2[k]))，利用流分布数据可得，此外还需要加上大流的个数。
应理解，在本申请的各种实施例中，上述各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请实施例的实施过程构成任何限定。
上文详细描述了根据本申请实施例的网络流测量的方法，在本申请中网络测量设备可以基于流关键字的比特级进行测量，记录测量信息；控制面设备根据网络设备上报的测量信息得到统计结果，从而实现可以灵活支持多种流的查询。应理解，本申请实施例的网络测量设备、控制面设备可以执行前述本申请实施例的各种方法，即以下各种产品的具体工作过程，可以参考前述方法实施例中的对应过程。
图9示出了根据本申请实施例的网络测量设备700的示意性框图(图9中的网络测量设备可以是图4中的任意一个节点)。该网络测量设备700可以对应于各方法实施例中网络测量设备，可以具有方法中的网络测量设备的任意功能。
如图9所示，该网络测量设备700包括通信模块710和处理模块720。
处理模块720，用于测量第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应。
通信模块710，用于向所述控制面设备发送所述第一数据。
在本申请的实施例中，网络测量设备通过第一数据包括的对应流的关键字的比特级的第一类数据结构进行流数据的测量，将基于第一数据记录的测量信息上报至控制面设备，控制面设备根据第一数据的概率分布特性，得到统计结果。从而实现了通过一种通用的测量方法，可以支持大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量的应用。
可选地，所述第一类数据结构包括对应流的关键字的至少一个比特的测量信息。
可选地，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第 二测量信息，所述第二测量信息与所述流的关键字对应。
可选地，所述第一测量信息与所述流的关键字的每个比特对应。
可选地，所述第一类数据结构包括对应所述关键字的每个比特的测量信息。
可选地，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
应理解，K个第一数据单元矩阵中的每一个数据单元矩阵与流的关键字的一个比特对应。
例如，流的关键字为4比特，则4个第一数据单元矩阵与流的关键字相对应，即关键字的第一比特与第一个第一数据单元矩阵对应，关键字的第二比特与第二个第一数据单元矩阵对应，第三比特与第三个第一数据单元矩阵对应，第四比特与第四个第一数据单元矩阵对应。
可选地，所述处理模块720，具体用于在所述第二数据单元矩阵中的M ij记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；在所述第一流关键字的第N比特的取值为预设值时，在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的所述第一测量信息，所述第一测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，第一测量信息和第二测量信息可以是统计值。
例如，所述处理模块720，具体用于在在所述第一流关键字的第N比特的取值为1时，在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，所述处理模块720，具体用于在在所述第一流关键字的第N比特的取值为0时，在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
可选地，所述第一流与M ij存在映射关系，所述映射关系为：
h i(key)＝j；
其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
需要说明的是，网络测量设备包括的处理模块可以是一个multi-sketch处理单元，multi-sketch可以是基于图6所示的数据结构进行测量信息的记录。通信模块可以是例如软件设备上实现一个函数，或硬件设备上一个算法单元，用于将multi-sketch记录的数据结构发送到控制面。
图10示出了根据本申请一个实施例的控制面设备800的示意性框图(图8中的控制面设备800可以是图4中控制面设备)。该控制面设备800可以对应于各方法实施例中的控制面设备，可以具有方法中的控制面设备的任意功能。
如图10所示，该控制面设备800可以包括通信模块810和处理模块820。
通信模块810，用于获取所述网络测量设备上报的第一数据，所述第一数据包括第一 类数据结构，所述第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应；
处理模块820，用于根据所述第一数据，得到统计结果。
在本申请的实施例中，控制面设备接收网络测量设备上报的测量信息，网络测量设备通过第一数据包括的对应流的关键字的比特级的第一类数据结构进行流数据的测量，将基于第一数据记录的测量信息上报至控制面设备，控制面设备根据第一数据的概率分布特性，得到统计结果。从而实现了通过一种通用的测量方法，可以支持大流、任意流大小、突变流等多种目标流量的测量，并且不受流分布的影响，实现采用单一传感器支持较广泛的网络流测量的应用。
可选地，所述第一类数据结构包括对应流的关键字的至少一个比特的测量信息。
可选地，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
可选地，所述第一类数据结构包括对应所述关键字的每个比特的测量信息。
可选地，所述第一测量信息与所述流的关键字的每个比特对应。
可选地，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
应理解，K个第一数据单元矩阵中的每一个数据单元矩阵与流的关键字的一个比特对应。
例如，流的关键字为4比特，则4个第一数据单元矩阵与流的关键字相对应，即关键字的第一比特与第一个第一数据单元矩阵对应，关键字的第二比特与第二个第一数据单元矩阵对应，第三比特与第三个第一数据单元矩阵对应，第四比特与第四个第一数据单元矩阵对应。
可选地，所述第二类数据结构中的M ij用于记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；
所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的所述第一测量信息，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述N为正整数。
例如，第一测量信息和第二测量信息可以是统计值。
可选地，所述第二类数据结构中的M ij用于记录第一流的统计值，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述统计值包括报文数量或报文长度中的至少一项；所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵 中的M ij用于在所述第一流关键字的第N比特的取值为1时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
例如，所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为0时，记录所述第一流的统计值，所述N小于或等于所述K，所述K与所述N均为正整数。
可选地，所述第一流与M ij存在映射关系，所述映射关系为：
h i(key)＝j；
其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
可选地，所述处理模块820，具体用于根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，所述第一集合包括从所述第一数据包括的r*c个栈stack中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一集合的残余流对应的所有的数据单元，所述r*c个栈中的任意一个栈为根据K+1个数据单元矩阵中的相同行和列的数据单元组成的数据单元组。
可选地，所述处理模块820，具体用于从所述r*c个栈stack中的第一栈中析取大流，所述第一集合包括所述第一栈中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一栈中析取大流的残余流对应的所有的数据单元。
可选地，若所述更新的所述第一数据满足正态分布，所述处理模块820，还用于确定所述第一集合包括所述第一数据中的所有大流。
可选地，若所述更新的所述第一数据不满足正态分布，所述处理模块820，还用于从所述第一数据包括的所述r*c个栈stack中析取大流。
可选地，若所述第一集合为空集，所述处理模块820，具体用于调整阈值后从所述第一数据包括的所述r*c个栈stack中析取大流，其中，所述阈值表示大流流量与总流量的比值或大流频度与总流量频度的比值。
可选地，所述处理模块820，具体用于获取所述r*c个栈stack中对应每比特的值为1的概率；根据所述概率确定第三集合，所述第三集合包括所述大流的候选关键字；根据所述第三集合确定所述大流的关键字。
可选地，所述处理模块820，具体用于基于每比特的所述第一类数据结构与所述第二类数据结构中相同栈的比值确定K个频度，计算出所述第三集合包括的至少一个关键字的频度；根据每比特确定性概率对所述至少一个关键字进行筛选，确定所述大流的关键字；根据哈希函数对所述大流的关键字进行频度校验。
可选地，所述统计结果包括所述第一数据中的大流列表，或所述第一数据中除大流外的残余流的数据，或所述第一数据中每比特的流数据的频度的分布中的至少一项。
可选地，所述通信模块810，还用于所述控制面设备接收流量查询请求；
所述处理模块820，还用于根据所述流量查询请求发送查询结果。
图11是根据本申请实施例提供的网络测量设备的结构框图。如图11所示，网络测量设备包括处理器801、存储器802和收发器803。处理器801可以用于对流数据进行处理，以及对网络测量设备进行控制，执行软件程序，处理软件程序的数据等。存储器802主要用于存储软件程序和数据。
当需要发送流数据时，处理器801对待发送的流数据进行处理后，输出至收发器803。 当有流数据发送到网络设备时，收发器803接收到该流数据并将该数据输出至处理器，处理器对该流数据进行处理。为便于说明，图8中仅示出了一个存储器和处理器。在实际的网络设备产品中，可以存在一个或多个处理器和一个或多个存储器。存储器也可以称为存储介质或者存储设备等。存储器可以是独立于处理器设置，也可以是与处理器集成在一起，本申请实施例对此不做限制。
在本申请实施例中，可以将具有收发功能的电路视为网络设备的收发器803，将具有处理功能的处理器视为网络测量设备的处理模块720。收发器也可以称为收发模块710、收发机、收发装置等。处理模块也可以称为处理器，处理单板，处理模块、处理装置等。
可选的，可以将收发器803中用于实现接收功能的器件视为接收单元，将收发器803中用于实现发送功能的器件视为发送单元，即收发器803包括接收单元和发送单元。接收单元有时也可以称为接收机、接收器、或接收电路等。发送单元有时也可以称为发射机、发射器或者发射电路等。
处理器801、存储器802和收发器803之间通过内部连接通路互相通信，传递控制和/或数据信号
上述本申请实施例揭示的方法可以应用于处理器801中，或者由处理器801实现。处理器801可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器801中的硬件的集成逻辑电路或者软件形式的指令完成。
本申请各实施例所述的处理器可以是通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的指令，结合其硬件完成上述方法的步骤。
可选的，在一些实施例中，存储器802可以存储用于执行如图5所示方法中网络测量设备执行的方法的指令。处理器801可以执行存储器802中存储的指令结合其他硬件(例如收发器803)完成如图5所示方法中网络测量设备执行的步骤，具体工作过程和有益效果可以参见图5所示实施例中的描述。
本申请实施例还提供一种芯片，该芯片包括收发单元和处理单元。其中，收发单元可以是输入输出电路、通信接口；处理单元为该芯片上集成的处理器或者微处理器或者集成电路。该芯片可以执行图5所示方法实施例中网络测量设备的方法。
本申请实施例还提供一种包含指令的计算机程序产品，该指令被执行时执行图4所示方法实施例中网络测量设备的方法。
本申请实施例还提供一种计算机可读存储介质，其上存储有指令，该指令被执行时执行图5所示方法实施例中网络测量设备的方法。
图12是根据本申请实施例提供的控制面设备的结构框图。如图12所示，控制面设备 包括处理器901、存储器902和收发器903。处理器901可以用于对流数据进行处理，以及对网络测量设备进行控制，执行软件程序，处理软件程序的数据等。存储器902主要用于存储软件程序和数据。
当需要发送流数据时，处理器901对待发送的流数据进行处理后，输出至收发器903。当有流数据发送到网络设备时，收发器903接收到该流数据并将该数据输出至处理器，处理器对该流数据进行处理。为便于说明，图12中仅示出了一个存储器和处理器。在实际的网络设备产品中，可以存在一个或多个处理器和一个或多个存储器。存储器也可以称为存储介质或者存储设备等。存储器可以是独立于处理器设置，也可以是与处理器集成在一起，本申请实施例对此不做限制。
在本申请实施例中，可以将具有收发功能的电路视为网络设备的收发器903，将具有处理功能的处理器视为控制面设备的处理模块820。收发器也可以称为通信模块810、收发机、收发装置等。处理模块也可以称为处理器，处理单板，处理模块、处理装置等。
可选的，可以将收发器903中用于实现接收功能的器件视为接收单元，将收发器903中用于实现发送功能的器件视为发送单元，即收发器903包括接收单元和发送单元。接收单元有时也可以称为接收机、接收器、或接收电路等。发送单元有时也可以称为发射机、发射器或者发射电路等。
处理器901、存储器902和收发器903之间通过内部连接通路互相通信，传递控制和/或数据信号
上述本申请实施例揭示的方法可以应用于处理器901中，或者由处理器901实现。处理器901可能是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤可以通过处理器901中的硬件的集成逻辑电路或者软件形式的指令完成。
本申请各实施例所述的处理器可以是通用处理器、数字信号处理器(digital signal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现成可编程门阵列(field programmable gate array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件。可以实现或者执行本申请实施例中的公开的各方法、步骤及逻辑框图。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。结合本申请实施例所公开的方法的步骤可以直接体现为硬件译码处理器执行完成，或者用译码处理器中的硬件及软件模块组合执行完成。软件模块可以位于随机存取存储器(random access memory，RAM)、闪存、只读存储器(read-only memory，ROM)、可编程只读存储器或者电可擦写可编程存储器、寄存器等本领域成熟的存储介质中。该存储介质位于存储器，处理器读取存储器中的指令，结合其硬件完成上述方法的步骤。
可选的，在一些实施例中，存储器902可以存储用于执行如图5所示方法中控制面设备执行的方法的指令。处理器901可以执行存储器902中存储的指令结合其他硬件(例如收发器903)完成如图5所示方法中控制面设备执行的步骤，具体工作过程和有益效果可以参见图5所示实施例中的描述。
本申请实施例还提供一种芯片，该芯片包括收发单元和处理单元。其中，收发单元可以是输入输出电路、通信接口；处理单元为该芯片上集成的处理器或者微处理器或者集成电路。该芯片可以执行图5所示方法实施例中控制面设备的方法。
本申请实施例还提供一种包含指令的计算机程序产品，该指令被执行时执行图5所示 方法实施例中控制面设备的方法。
本申请实施例还提供一种计算机可读存储介质，其上存储有指令，该指令被执行时执行图5所示方法实施例中控制面设备的方法。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(read-only memory，ROM)、随机存取存储器(random access memory，RAM)、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。

Claims (30)

  1. 一种网络流测量的方法，其特征在于，所述方法适用于包括网络测量设备和控制面设备的系统中，所述方法包括：
    所述网络测量设备测量第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应；
    所述网络测量设备向所述控制面设备发送所述第一数据。
  2. 根据权利要求1所述的方法，其特征在于，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
  3. 根据权利要求1或2所述的方法，其特征在于，所述第一测量信息与所述流的关键字的每个比特对应。
  4. 根据权利要求2或3所述的方法，其特征在于，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
  5. 根据权利要求4所述的方法，其特征在于，所述方法还包括：
    所述网络测量设备在所述第二数据单元矩阵中的M ij记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；
    在所述第一流关键字的第N比特的取值为预设值时，所述测量设备在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的所述第一测量信息，所述第一测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述N为正整数。
  6. 根据权利要求5所述的方法，其特征在于，所述第一流与M ij存在映射关系，所述映射关系为：
    h i(key)＝j；
    其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
  7. 一种网络流测量的方法，其特征在于，所述方法适用于包括网络测量设备和控制面设备的系统中，所述方法包括：
    所述控制面设备获取所述网络测量设备上报的第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应；
    所述控制面设备根据所述第一数据，得到统计结果。
  8. 根据权利要求7所述的方法，其特征在于，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
  9. 根据权利要求7或8所述的方法，其特征在于，所述第一测量信息与所述流的关键字的每个比特对应。
  10. 根据权利要求8或9所述的方法，其特征在于，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
  11. 根据权利要求10所述的方法，其特征在于，所述第二类数据结构中的M ij用于记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；
    所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的所述第一测量信息，所述第一测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述N为正整数。
  12. 根据权利要求11所述的方法，其特征在于，所述第一流与M ij存在映射关系，所述映射关系为：
    h i(key)＝j；
    其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
  13. 根据权利要求10至12中任一项所述的方法，其特征在于，所述控制面设备根据所述第一数据，得到统计结果，包括：
    所述控制面设备根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，所述第一集合包括从所述第一数据包括的r*c个栈stack中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一集合的残余流所对应的所有的数据单元，所述r*c个栈中的任意一个栈为根据K+1个数据单元矩阵中的相同行和列的数据单元组成的数据单元组。
  14. 根据权利要求13所述的方法，其特征在于，所述控制面设备根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，包括：
    所述控制面设备从所述r*c个栈stack中的第一栈中析取大流，所述第一集合包括所述第一栈中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一栈中析取大流的残余流对应的所有的数据单元。
  15. 根据权利要求13或14所述的方法，其特征在于，所述方法还包括：
    若所述更新的所述第一数据满足正态分布，则确定所述第一集合包括所述第一数据中的所有大流；或
    若所述更新的所述第一数据不满足正态分布，则从所述第一数据包括的所述r*c个栈stack中析取大流。
  16. 一种网络测量设备，其特征在于，所述网络测量设备适用于包括所述网络测量设备和控制面设备的系统中，所述网络测量设备包括：
    处理模块，用于测量第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括流的第一测量信息，所述第一测量信息与所述流的关键字的比特对应；
    通信模块，用于向所述控制面设备发送所述第一数据。
  17. 根据权利要求16所述的网络测量设备，其特征在于，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
  18. 根据权利要求16或17所述的网络测量设备，其特征在于，所述第一测量信息与所述流的关键字的每个比特对应。
  19. 根据权利要求17或18所述的网络测量设备，其特征在于，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息，所述K为大于或等于1的正整数。
  20. 根据权利要求19所述的网络测量设备，其特征在于，所述处理模块，具体用于在所述第二数据单元矩阵中的M ij记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；
    在所述第一流的关键字的第N比特的取值为预设值时，在所述第一类数据结构中的第N个数据单元矩阵的M ij记录所述第一流的所述第一测量信息，所述第一测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述N为正整数。
  21. 根据权利要求20所述的网络测量设备，其特征在于，所述第一流与M ij存在映射关系，所述映射关系为：
    h i(key)＝j；
    其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
  22. 一种控制面设备，其特征在于，所述控制面设备适用于包括网络测量设备和所述控制面设备的系统中，所述控制面设备包括：
    通信模块，用于获取所述网络测量设备上报的第一数据，所述第一数据包括第一类数据结构，所述第一类数据结构包括第一测量信息，所述第一测量信息与所述流的关键字的比特对应；
    处理模块，根据所述第一数据，得到统计结果。
  23. 根据权利要求22所述的控制面设备，其特征在于，所述第一数据还包括第二类数据结构，所述第二类数据结构包括所述流的第二测量信息，所述第二测量信息与所述流的关键字对应。
  24. 根据权利要求22或23所述的控制面设备，其特征在于，所述第一测量信息与所述流的关键字的每个比特对应。
  25. 根据权利要求23或24所述的控制面设备，其特征在于，所述第一类数据结构为K个第一数据单元矩阵，其中，每个第一数据单元矩阵用于根据所述流的关键字的一个比特的取值记录所述第一测量信息；所述第二类数据结构为第二数据单元矩阵，所述第二数据单元矩阵用于记录所述第二测量信息。
  26. 根据权利要求25所述的控制面设备，其特征在于，所述第二类数据结构中的M ij用于记录第一流的所述第二测量信息，其中，M ij表示第i行第j列的数据单元，所述i为 小于或等于r的正整数，所述j为小于或等于c的正整数，所述第二测量信息包括所述第一流中的报文数量或报文长度中的至少一项；
    所述第一类数据结构为K个第一数据单元矩阵，其中，第N个数据单元矩阵中的M ij用于在所述第一流关键字的第N比特的取值为预设值时，记录所述第一流的所述第一测量信息，所述第一测量信息包括所述第一流中的报文数量或报文长度中的至少一项，所述N小于或等于所述K，所述N为正整数。
  27. 根据权利要求26所述的控制面设备，其特征在于，所述第一流与M ij存在映射关系，所述映射关系为：
    h i(key)＝j；
    其中，h i()表示第i个哈希函数，所述key表示所述第一流对应的关键字。
  28. 根据权利要求25至27中任一项所述的控制面设备，其特征在于，所述处理模块，具体用于根据所述关键字的每比特上的分布特性获取第一集合和更新的所述第一数据，所述第一集合包括从所述第一数据包括的r*c个栈stack中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一集合的残余流所对应的所有的数据单元，所述r*c个栈中的任意一个栈为根据K+1个数据单元矩阵中的相同行和列的数据单元组成的数据单元组。
  29. 根据权利要求28所述的控制面设备，其特征在于，所述处理模块，具体用于从所述r*c个栈stack中的第一栈中析取大流，所述第一集合包括所述第一栈中析取的大流，所述更新的所述第一数据包括所述r*c个栈stack中除所述第一栈中析取大流的残余流对应的所有的数据单元。
  30. 根据权利要求28或29所述的控制面设备，其特征在于，所述处理模块还用于，若所述更新的所述第一数据满足正态分布，则确定所述第一集合包括所述第一数据中的所有大流；或
    若所述更新的所述第一数据不满足正态分布，则从所述第一数据包括的所述r*c个栈stack中析取大流。
PCT/CN2019/097092 2018-07-27 2019-07-22 网络流测量的方法、网络测量设备以及控制面设备 Ceased WO2020020098A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP19840256.2A EP3823217A4 (en) 2018-07-27 2019-07-22 NETWORK FLOW MEASUREMENT PROCESS, NETWORK MEASURING DEVICE AND CONTROL PLAN DEVICE
US17/158,586 US11706114B2 (en) 2018-07-27 2021-01-26 Network flow measurement method, network measurement device, and control plane device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201810844967.6A CN110768856B (zh) 2018-07-27 2018-07-27 网络流测量的方法、网络测量设备以及控制面设备
CN201810844967.6 2018-07-27

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/158,586 Continuation US11706114B2 (en) 2018-07-27 2021-01-26 Network flow measurement method, network measurement device, and control plane device

Publications (1)

Publication Number Publication Date
WO2020020098A1 true WO2020020098A1 (zh) 2020-01-30

Family

ID=69180214

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/097092 Ceased WO2020020098A1 (zh) 2018-07-27 2019-07-22 网络流测量的方法、网络测量设备以及控制面设备

Country Status (4)

Country Link
US (1) US11706114B2 (zh)
EP (1) EP3823217A4 (zh)
CN (1) CN110768856B (zh)
WO (1) WO2020020098A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117336209A (zh) * 2023-10-14 2024-01-02 东南大学 基于通用略图的高速网络逐流基数和分布函数的估算方法
CN120378339A (zh) * 2025-06-26 2025-07-25 苏州小棉袄信息技术股份有限公司 面向高速数据流实时分析的突发流量监测方法、装置、终端及存储介质

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111541617B (zh) * 2020-04-17 2021-11-02 网络通信与安全紫金山实验室 一种用于高速大规模并发数据流的数据流表处理方法及装置
CN114124731B (zh) * 2020-08-26 2023-03-31 华为技术有限公司 流量监控方法、装置、集成电路及网络设备
EP4195609B1 (en) 2020-08-26 2025-08-27 Huawei Technologies Co., Ltd. Traffic monitoring method and apparatus, integrated circuit, network device, and network system
CN114584493A (zh) * 2020-11-30 2022-06-03 华为技术有限公司 数据流测量的方法和装置
CN115242668A (zh) * 2021-04-23 2022-10-25 华为技术有限公司 一种网络数据的处理方法及装置
CN113839835B (zh) * 2021-09-27 2023-09-26 长沙理工大学 一种基于小流过滤的Top-k流精确监控系统
CN119781687B (zh) * 2024-12-25 2025-10-10 武汉大学 一种基于冷热流分离的网络流数据的存储方法及系统、插入方法、查询方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060075489A1 (en) * 2004-09-30 2006-04-06 Lucent Technologies, Inc. Streaming algorithms for robust, real-time detection of DDoS attacks
CN101227318A (zh) * 2007-12-04 2008-07-23 东南大学 高速网络流量的超点实时检测方法
CN103647670A (zh) * 2013-12-20 2014-03-19 北京理工大学 一种基于sketch的数据中心网络流量分析方法
CN107566206A (zh) * 2017-08-04 2018-01-09 华为技术有限公司 一种流量测量方法、设备及系统

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2884696A1 (en) * 2013-12-11 2015-06-17 Telefonaktiebolaget L M Ericsson (publ) Sketch based monitoring of a communication network
CN104657450B (zh) * 2015-02-05 2018-09-25 中国科学院信息工程研究所 面向大数据环境的概要信息动态构建与查询方法及装置
US10992568B2 (en) * 2017-01-31 2021-04-27 Vmware, Inc. High performance software-defined core network
US11044202B2 (en) * 2017-02-06 2021-06-22 Silver Peak Systems, Inc. Multi-level learning for predicting and classifying traffic flows from first packet data
US20190114319A1 (en) * 2017-10-17 2019-04-18 Oracle International Corporation Streaming latent dirichlet allocation
WO2019168965A1 (en) * 2018-03-01 2019-09-06 Huawei Technologies Co., Ltd. Packet-programmable statelets
US10833749B1 (en) * 2019-07-01 2020-11-10 Rampart Communications, Inc. Communication system and method using layered construction of arbitrary unitary matrices

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060075489A1 (en) * 2004-09-30 2006-04-06 Lucent Technologies, Inc. Streaming algorithms for robust, real-time detection of DDoS attacks
CN101227318A (zh) * 2007-12-04 2008-07-23 东南大学 高速网络流量的超点实时检测方法
CN103647670A (zh) * 2013-12-20 2014-03-19 北京理工大学 一种基于sketch的数据中心网络流量分析方法
CN107566206A (zh) * 2017-08-04 2018-01-09 华为技术有限公司 一种流量测量方法、设备及系统

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3823217A4

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117336209A (zh) * 2023-10-14 2024-01-02 东南大学 基于通用略图的高速网络逐流基数和分布函数的估算方法
CN120378339A (zh) * 2025-06-26 2025-07-25 苏州小棉袄信息技术股份有限公司 面向高速数据流实时分析的突发流量监测方法、装置、终端及存储介质

Also Published As

Publication number Publication date
CN110768856B (zh) 2022-01-14
US20210152454A1 (en) 2021-05-20
US11706114B2 (en) 2023-07-18
EP3823217A1 (en) 2021-05-19
EP3823217A4 (en) 2021-08-04
CN110768856A (zh) 2020-02-07

Similar Documents

Publication Publication Date Title
US11706114B2 (en) Network flow measurement method, network measurement device, and control plane device
CN108076040B (zh) 一种基于杀伤链和模糊聚类的apt攻击场景挖掘方法
US9979624B1 (en) Large flow detection for network visibility monitoring
CN112434298B (zh) 一种基于自编码器集成的网络威胁检测系统
CN106657038B (zh) 一种基于对称度Sketch的网络流量异常检测与定位方法
US10097464B1 (en) Sampling based on large flow detection for network visibility monitoring
US10536360B1 (en) Counters for large flow detection
CN112600792B (zh) 一种物联网设备的异常行为检测方法及系统
US20110149745A1 (en) Detecting and classifying anomalies in communication networks
JP2020529171A (ja) トラフィック測定方法、デバイス、およびシステム
CN107683586A (zh) 用于异常检测中的计算基于小区密度的稀有度的方法和装置
EP4075749B1 (en) Detection method and detection device for heavy flow data stream
US10003515B1 (en) Network visibility monitoring
CN115102716B (zh) 一种基于自适应采样阈值的网络大流检测方法及系统
He et al. Histsketch: A compact data structure for accurate per-key distribution monitoring
CN110071934B (zh) 用于网络异常检测的局部敏感性计数摘要方法及系统
CN116170281A (zh) 告警关联规则生成方法、装置、电子设备和存储介质
CN115473688B (zh) 面向软件定义网络的异常检测方法、装置及设备
CN111817971A (zh) 一种基于深度学习的数据中心网络流量拼接方法
JPWO2015182629A1 (ja) 監視システム、監視装置及び監視プログラム
CN109952743A (zh) 用于低内存和低流量开销大流量对象检测的系统和方法
CN106372171A (zh) 监控平台实时数据处理方法
CN116318800A (zh) 一种bgp路由数据的监测方法、装置、及电子设备
Wang et al. Inferring higher-order structure statistics of large networks from sampled edges
Sun et al. DDoS attack detection combining time series-based multi-dimensional sketch and machine learning

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19840256

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE