WO2020228489A1 - 基于视觉的触觉测量方法、装置、芯片及存储介质 - Google Patents

基于视觉的触觉测量方法、装置、芯片及存储介质 Download PDF

Info

Publication number
WO2020228489A1
WO2020228489A1 PCT/CN2020/085608 CN2020085608W WO2020228489A1 WO 2020228489 A1 WO2020228489 A1 WO 2020228489A1 CN 2020085608 W CN2020085608 W CN 2020085608W WO 2020228489 A1 WO2020228489 A1 WO 2020228489A1
Authority
WO
WIPO (PCT)
Prior art keywords
sample
neural network
curvature
feedforward neural
estimation model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/085608
Other languages
English (en)
French (fr)
Inventor
郑宇�
许忠锦
张正友
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to EP20804776.1A priority Critical patent/EP3971684B1/en
Publication of WO2020228489A1 publication Critical patent/WO2020228489A1/zh
Priority to US17/369,837 priority patent/US12214487B2/en
Anticipated expiration legal-status Critical
Priority to US19/003,999 priority patent/US20260115934A1/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
    • B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
    • B25J13/00—Controls for manipulators
    • B25J13/08—Controls for manipulators by means of sensing devices, e.g. viewing or touching devices
    • B25J13/081—Touching devices, e.g. pressure-sensitive
    • B25J13/084—Tactile sensors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B5/00—Measuring arrangements characterised by the use of mechanical techniques
    • G01B5/20—Measuring arrangements characterised by the use of mechanical techniques for measuring contours or curvatures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/40—Analysis of texture
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/64—Analysis of geometric attributes of convexity or concavity
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
    • G06V10/225—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition based on a marking or identifier characterising the area
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/758—Involving statistics of pixels or of feature values, e.g. histogram matching
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776—Validation; Performance evaluation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes

Definitions

  • This application relates to the field of human-computer interaction, and in particular to a vision-based tactile measurement method, device, chip and storage medium.
  • the tactile sensor is a sensor used to imitate the function of the tactile sense, which can measure the tactile sense of the contacting object, such as the contact position and contact force.
  • tactile sensors are mostly used in the robotics field.
  • a tactile sensor is provided.
  • the tactile sensor is provided with a semicircular flexible sensing surface.
  • the inner surface of the flexible sensing surface is provided with a plurality of marking points arranged in an array and an image arranged toward the inner surface.
  • Sensing components After the outer surface of the flexible sensing surface is in contact with the object, the flexible sensing surface will be deformed, causing multiple marking points on the inner surface to change positions due to the deformation.
  • the image sensing component collects the inner surface image of the flexible sensing surface, and transmits the inner surface image to the chip.
  • a convolutional neural network (CNN) is set in the chip, and the inner surface image is processed through the convolutional neural network to obtain the analysis result of the contact force.
  • the training process of the above-mentioned convolutional neural network is relatively complicated, and as many as 20,000 training samples are required to achieve better training results.
  • Various embodiments of the present application provide a vision-based tactile measurement method, device, chip, and storage medium, a feedforward neural network training method, device, computer equipment and storage medium, and a tactile sensor system , And a robotic system.
  • a vision-based tactile measurement method is executed by a chip, the chip is connected to a tactile sensor, the tactile sensor includes a sensing surface and an image sensing component, the sensing surface is provided with a marking pattern;
  • the method includes: acquiring an image sequence collected by the image sensing component on the sensing surface, the image of the image sequence includes the mark pattern; and according to the image sequence in the adjacent images in the image sequence Mark patterns, calculate the difference features of the mark patterns; and call a feedforward neural network to process the difference characteristics of the mark patterns to obtain a tactile measurement result; wherein the number of hidden layers in the feedforward neural network is less than a threshold.
  • a method for training a feedforward neural network which is executed by a computer device, and the method includes:
  • the training samples including a sample image sequence and sample tactile results, the sample image sequence being an image sequence collected by an image sensing component in a tactile sensor, the tactile sensor including a sensing surface and an image sensing component,
  • the sensing surface is provided with a marking pattern, and the images of the image sequence include the marking pattern; according to the position of the marking pattern in adjacent images in the sample image sequence, a sample of the marking pattern is calculated Difference feature; call the feedforward neural network to process the sample difference feature of the mark pattern to obtain the predicted haptic result; the number of hidden layers in the feedforward neural network is less than the threshold; compare the predicted haptic result and the sample Error calculation is performed on the haptic result to obtain an error loss; and the feedforward neural network is trained according to the error loss through an error back propagation algorithm to obtain a trained feedforward neural network.
  • a vision-based tactile measurement device the device is applied to a chip, the chip is connected to a tactile sensor, the tactile sensor includes a sensing surface and an image sensing component, the sensing surface is provided with a marking pattern;
  • the device includes: a first acquisition module, configured to acquire an image sequence collected by the image sensing component on the sensing surface, the image of the image sequence includes the marking pattern; a first calculation module, Calculating the difference characteristics of the marking patterns according to the marking patterns in the adjacent images in the image sequence; and a feedforward neural network for processing the difference characteristics of the marking patterns to obtain a tactile measurement result; Wherein, the number of hidden layers in the feedforward neural network is less than a threshold.
  • a training device for a feedforward neural network comprising:
  • the second acquisition module is used to acquire training samples, the training samples include a sample image sequence and sample tactile results, the sample image sequence is an image sequence collected by an image sensing component in a tactile sensor, and the tactile sensor includes a flexible sensor.
  • a sensing surface and an image sensing component disposed toward the inner surface of the flexible sensing surface, the flexible sensing surface is provided with a marking pattern, and the images of the image sequence include the position of the marking pattern;
  • a second calculation The module is used to calculate the sample difference feature of the marking pattern according to the position of the marking pattern in the adjacent images in the sample image sequence;
  • the feedforward neural network model is used to calculate the sample difference of the marking pattern
  • the feature is processed to obtain a predicted haptic result, and the number of hidden layers in the feedforward neural network is less than a threshold; an error calculation module is used to perform error calculation on the predicted haptic result and the sample haptic result to obtain an error loss;
  • a training module which is used to train the feedforward neural network according to the error loss through an
  • a tactile sensor system comprising: a tactile sensor and a chip, the tactile sensor comprising a sensing surface and an image sensing component, the sensing surface is provided with a marking pattern, the image sensing component and the chip Connected; the chip includes at least one of a programmable logic circuit and program instructions, and when the chip is running, the chip is used to perform the visual-based tactile measurement method as described in the above aspect.
  • a computer device includes a memory and a processor.
  • the memory stores computer readable instructions.
  • the processor executes the computer readable instructions, the steps of the above-mentioned feedforward neural network training method are realized.
  • a computer-readable storage medium that stores computer-readable instructions that, when executed by a processor, implement the steps of the above-mentioned vision-based tactile measurement method, or the above-mentioned feedforward neural network training method step.
  • a robot system comprising: a chip and a tactile sensor, the tactile sensor is arranged at least one of a fingertip part and a skin part, the tactile sensor includes a sensing surface and an image sensing component, the The sensing surface is provided with a marking pattern, and the image sensing component is connected to the chip; the chip includes at least one of a programmable logic circuit and a program instruction, and when the chip is running, the chip is used to execute the above The vision-based tactile measurement method described in the aspect.
  • FIG. 1 is a schematic structural diagram of a tactile sensor in related technology provided by an exemplary embodiment of the present application
  • Fig. 2 is a schematic structural diagram of a touch sensor system provided by an exemplary embodiment of the present application
  • Fig. 3 is a schematic diagram of a flexible sensing surface provided by an exemplary embodiment of the present application.
  • Fig. 4 is a flowchart of a method for using a touch sensor provided by an exemplary embodiment of the present application
  • FIG. 5 is a schematic diagram of displacement recording of marking points provided by an exemplary embodiment of the present application.
  • Fig. 6 is a schematic structural diagram of a feedforward neural network provided by an exemplary embodiment of the present application.
  • FIG. 7 is a flow chart of a method for using a feedforward neural network provided by an exemplary embodiment of the present application.
  • FIG. 8 is a schematic diagram of each model structure in a feedforward neural network provided by an exemplary embodiment of the present application.
  • Fig. 9 is a flowchart of a method for using a position estimation model provided by an exemplary embodiment of the present application.
  • Fig. 10 is a flowchart of a method for using a contact force estimation model provided by another exemplary embodiment of the present application.
  • Fig. 11 is a flowchart of a method for using a surface classification model and a curvature estimation model provided by an exemplary embodiment of the present application;
  • FIG. 12 is a flowchart of a method for using a surface classification model provided by an exemplary embodiment of the present application.
  • FIG. 13 is a flowchart of a method for using a curvature estimation model of a spherical surface provided by an exemplary embodiment of the present application;
  • FIG. 14 is a flowchart of a method for using a curvature estimation model of a cylindrical surface provided by an exemplary embodiment of the present application.
  • FIG. 15 is a flowchart of a method for calculating the displacement of marker points on an image array provided by an exemplary embodiment of the present application
  • Fig. 16 is a flowchart of a visual-based tactile measurement training method provided by an exemplary embodiment of the present application.
  • FIG. 17 is a flowchart of a method for training a position estimation model provided by an exemplary embodiment of the present application.
  • FIG. 18 is a flowchart of a training method of a contact force estimation model provided by an exemplary embodiment of the present application.
  • Fig. 19 is a flowchart of a training method representing a surface classification model provided by an exemplary embodiment of the present application.
  • FIG. 20 is a flowchart of a training method of a spherical estimation model provided by an exemplary embodiment of the present application.
  • FIG. 21 is a flowchart of a method for training a cylindrical surface estimation model provided by another exemplary embodiment of the present application.
  • FIG. 22 is a schematic structural diagram of models other than the surface classification model provided by an exemplary embodiment of the present application.
  • FIG. 23 is a schematic diagram of a model structure of a surface classification model provided by an exemplary embodiment of the present application.
  • FIG. 24 is a schematic diagram of a sample curved surface for training a feedforward neural network provided by an exemplary embodiment of the present application.
  • FIG. 25 is a table of the number of used training sample surfaces provided by an exemplary embodiment of the present application.
  • FIG. 26 is a schematic diagram of a geometric model of a tactile sensor provided by an exemplary embodiment of the present application.
  • FIG. 27 is a schematic diagram of training results of a position estimation model provided by an exemplary embodiment of the present application.
  • FIG. 28 is a schematic diagram of training results of a contact force estimation model provided by an exemplary embodiment of the present application.
  • FIG. 29 is a confusion matrix representing the accuracy of training results of a surface classification model provided by an exemplary embodiment of the present application.
  • FIG. 30 is a schematic diagram of training results of a spherical estimation model provided by an exemplary embodiment of the present application.
  • FIG. 31 is a schematic diagram of training results of a cylindrical surface estimation model provided by an exemplary embodiment of the present application.
  • Fig. 32 is a flowchart of a method of using the touch sensor system provided by an exemplary embodiment of the present application.
  • Fig. 33 is a block diagram of a vision-based tactile measurement device provided by an exemplary embodiment of the present application.
  • Fig. 34 is a block diagram of a training device for a feedforward neural network provided by another exemplary embodiment of the present application.
  • Fig. 35 is a block diagram of a computer device provided by an exemplary embodiment of the present application.
  • Feedforward Neural Network an artificial neural network with a unidirectional structure.
  • the feedforward neural network includes at least two neural network layers. Among them, each neural network layer contains several neurons, and each neuron is arranged hierarchically. There is no interconnection between neurons in the same layer, and the transmission of information between layers is only carried out in one direction.
  • Logistic (Sigmoid) function It is an "S"-shaped function used to describe the growth trend as roughly exponential growth at the initial stage; then as it becomes saturated, the increase slows down; finally, it increases when it reaches maturity The process of stopping.
  • Softmax Normalized exponent
  • Hidden layer a neural network layer structure used to input or analyze data.
  • Output layer A neural network layer structure used to output results.
  • a tactile sensor is provided in the related art, as shown in FIG. 1.
  • the tactile sensor includes: a silicone sensor surface 11, marking points 12 arranged on the inner surface of the silicone sensor surface 11, a model front end 13 of the tactile sensor made by 3D printing technology, and the model front end 13 is used to fix the silicone sensor surface 11.
  • LED light emitting diodes
  • This technology uses a probabilistic model based on Bayesian theorem to distinguish the position of the contact point, the radius of curvature and the direction of the contact edge, and the convolutional neural network model (CNN) algorithm is used in the latest research.
  • CNN convolutional neural network model
  • a feedforward neural network is used to provide a vision-based tactile measurement solution.
  • This method uses the displacement of the marked points on two consecutive images in the image array as feature values and inputs them to the feedforward neural network to obtain the position of the contact point, the magnitude and/or direction of the contact force, and the radius of curvature of the contact surface.
  • CNN convolutional neural network model
  • this method simplifies the input feature value (only the displacement and/or deformation of the marker point is required, not the entire image).
  • the training sample Significantly reduce, improve the training efficiency of the neural network, so as to meet the requirements of simplifying the method of using tactile sensors, and without a large number of samples training, the feedforward neural network can achieve the same effect (or better) demand.
  • the vision-based tactile measurement method is applied to the chip 117, which can be any one of a CPU, a GPU, a neural network chip, a microprocessor, or an FPGA circuit. It is not limited, the chip 117 is connected to a tactile sensor, and the tactile sensor includes a sensing surface 111 and an image sensing component 115, and the sensing surface 111 is provided with a marking pattern 112.
  • the sensing surface 111 is a flexible sensing surface that can be deformed when in contact with other objects.
  • the image sensing component 115 may be a camera. The camera can be arranged facing the inner surface of the sensing surface 111.
  • Fig. 2 shows a schematic structural diagram of a tactile sensor system 300 provided by an exemplary embodiment of the present application.
  • the tactile sensor includes a sensing surface 111, a base 113, a column 114, and an image sensing component 115 disposed toward the inner surface of the sensing surface 111, and a bottom plate 116 is used to place the tactile sensor.
  • FIG. 3 shows a schematic diagram of the inner surface of a sensing surface 111 provided by an exemplary embodiment of the present application, and the inner surface of the sensing surface 111 is provided with a marking pattern 112.
  • the shape of the sensing surface 111 is not limited.
  • the sensing surface 111 can be any one of rectangular, hexagonal, circular, elliptical, hemispherical, or planar shapes.
  • a hemispherical flexible sensing surface 111 is used as an example for description, as shown in FIG. 3.
  • the marking pattern is implemented by using at least two marking points, or by using a grid, or by using both marking points and a grid.
  • a grid is a pattern with intersecting grid lines, and the intersecting grid lines form grid points.
  • marking points 112 are provided on the inner surface (or inside) of the sensing surface 111, and the marking points 112 may be arranged in an array or non-array arrangement, for example, a 4 ⁇ A rectangular array of 4 or 6 ⁇ 6, or a circular non-array arrangement.
  • the distance between adjacent marking points 112 may be equal or unequal. When the distance between adjacent marking points 112 is equal, the displacement of the marking point 112 changes uniformly.
  • the marking points 112 can be centrally arranged on the inner surface of the sensing surface 111, for example, a 4 ⁇ 4 rectangular array of marking points 112 are provided on the sensing surface 111, or they can be arranged along the edge of the sensing surface 111 .
  • the color of the marking point can be any color. In this application, the black marking point 112 is selected to distinguish it from the white sensing surface 111, which can better indicate the displacement of the marking point 112.
  • the array of black marking points 112 is centrally arranged on the inner surface of the sensing surface 111, and the distance between the marking points 112 on the edge of the marking point array and each edge on the sensing surface 111 is equal to each other. The distances between adjacent marking points 112 are equal.
  • a rectangular array with a circular sensing surface 111 and a 6 ⁇ 6 marking point 112 is used as an example for description, as shown in FIG. 3.
  • Fig. 4 shows a flowchart of a vision-based tactile measurement method provided by an exemplary embodiment of the present application. The method may be executed by the chip 117 in Fig. 1, and the method includes:
  • Step 201 Obtain an image sequence collected by the image sensing component on the inner surface, and the image of the image sequence includes a marking pattern.
  • the sensing surface 111 of the touch sensor When the touched object contacts the sensing surface 111 of the touch sensor, the sensing surface 111 is deformed, and the image sensing component continuously photographs the inner surface of the sensing surface 111 at a certain frequency and transmits images to the chip, thus forming an image array.
  • the aforementioned frequency can be set according to the displacement of the marking point 112, such as 30 frames/sec or 60 frames/sec.
  • the tactile sensor in the high-frequency shooting state of the image sensor component 115, the tactile sensor can also detect the slippage of the contacted object, and can even detect a large sudden force.
  • Step 202 Calculate the difference characteristics of the marked images according to the marked images in the adjacent images in the image sequence.
  • adjacent images in the image array are two adjacent images.
  • the difference feature of the marking pattern includes at least one of the displacement and deformation of the marking point, such as including the displacement and deformation of the marking point.
  • the difference feature of the marking pattern includes at least one of the displacement of the grid points in the grid and the deformation of the grid lines, such as the displacement of the grid points in the grid, or the grid The displacement of the grid points and the deformation of the grid lines.
  • the above chip marks the same mark according to the two closest mark points 112 in two adjacent images, and can track the movement of each mark point, that is, calculate the displacement of the mark point 112 As a difference feature, as shown in Figure 5.
  • Step 203 Invoke the feedforward neural network to process the difference characteristics of the marking pattern, and obtain the tactile measurement result.
  • the feedforward neural network can be one or more neural network models, and each neural network model corresponds to a different function.
  • the number of hidden layers in the feedforward neural network is less than the threshold. In some embodiments, the threshold is 2.
  • the hidden layer and output layer of each model in the feedforward neural network are called, and the difference characteristics of the marking pattern are processed according to the models that realize different functions in the feedforward neural network, and the tactile measurement results are obtained.
  • the number of neural network models, hidden layers, and output layers with different functions can be designed according to measurement needs.
  • the number of hidden layers and output layers is 1 respectively for description.
  • the method provided by the embodiments of the present application uses the difference feature of the mark pattern as the input feature, compared with the use of an image as the input feature in the related technology, can reduce the number of input features, thereby reducing the amount of calculation; at the same time, By using a feedforward neural network with hidden layers less than a threshold for feature extraction and prediction, compared with a CNN network with a larger number of layers, it can use less calculation to predict similar or better tactile measurement results .
  • Fig. 6 shows a schematic structural diagram of a feedforward neural network 200 provided by an exemplary embodiment.
  • the feedforward neural network 200 is provided with a hidden layer 202 and an output layer 203.
  • the method for the above-mentioned feedforward neural network 200 to process the difference features of the marking pattern 112 to obtain the measurement result includes the following steps, as shown in FIG. 7:
  • step 301 the hidden layer 201 in the feedforward neural network 200 is called to perform feature extraction on the difference features of the mark pattern 112 to obtain a feature representation.
  • step 302 the output layer 202 in the feedforward neural network is called to process the feature representation to obtain the tactile measurement result.
  • n hidden neurons in the hidden layer 201 There are n hidden neurons in the hidden layer 201, and n is an integer.
  • the hidden layer 201 is constructed based on Sigmoid hidden neurons
  • the aforementioned output layer 202 is constructed based on Softmax output neurons or linear output neurons.
  • the number of neurons in the hidden layer can be any integer greater than zero, and the number of input feature values can also be any integer greater than zero.
  • the above-mentioned neurons can be designed according to different functions. .
  • the marking pattern 112 is implemented by using 36 marking points. Taking 100 hidden neurons in the hidden layer 201 as an example, 72 features are input to the input layer 201 of the feedforward neural network 200. The characteristic value is the displacement of 36 mark points 112 (x1, y1, x2, y2,..., x36, y36).
  • the structure of the tactile sensor involved in this application is relatively simple, and at the same time, the method of inputting the displacement of the marker point on the continuous image array as the feature value into the neural network model is adopted. And design a feedforward neural network with a simple hidden layer to measure the touched objects, which simplifies the measurement process and reduces the number of training samples.
  • the hidden layer in the structure of the feedforward neural network transmits information in a unidirectional manner, and the hidden layer contains at least one.
  • the three-dimensional information of the contact force includes the magnitude and/or direction of the contact force
  • the above-mentioned feedforward neural network includes: a position estimation model for estimating the position of the contact point, a contact force estimation model for estimating the three-dimensional information of the contact force (the magnitude and/or direction of the contact force), and a surface for classifying the contact surface
  • a position estimation model for estimating the position of the contact point
  • a contact force estimation model for estimating the three-dimensional information of the contact force (the magnitude and/or direction of the contact force)
  • a surface for classifying the contact surface The classification model and the curvature estimation model used to estimate the local radius of curvature of the contact surface are shown in Figure 8.
  • the tactile measurement result includes the contact position
  • the feedforward neural network includes: a position estimation model
  • the position estimation model includes a first hidden layer and a first output
  • the measurement method of layer and contact position is shown in Figure 9, including:
  • Step 401 Invoke the first hidden layer in the position estimation model to perform feature extraction on the difference features of the marking pattern to obtain a feature representation of the contact position;
  • the first hidden layer is used for feature extraction of the displacement of the input marking points.
  • the first hidden layer is based on Sigmoid Constructed by hidden neurons, the above-mentioned feature representation of the contact position is a representation of the feature representation corresponding to the contact position in the form of a vector. It has been explained in step 301 of using the feedforward neural network described above, and will not be repeated here.
  • Step 402 Invoke the first output layer in the position estimation model to process the contact position feature representation to obtain the contact position.
  • step 302 of using the feedforward neural network described above has been described in step 302 of using the feedforward neural network described above, and will not be repeated here.
  • the number of hidden layers and output layers in the above position estimation model are both integers greater than zero, and the above neurons can be selected according to the realization of different functions.
  • This application uses the first hidden layer and the first output The number of layers is 1, and the neurons are respectively selected based on Sigmoid hidden neurons and linear output neurons as examples, as shown in Figure 22.
  • the feature value is input to the feedforward neural network 200, and the Sigmoid hidden neuron in the first hidden layer in the position estimation model is called to process the above feature value to obtain the feature representation of the contact position; the feature representation of the contact position will be used as The input value is input to the first output layer, and the linear output neuron in the first output layer will perform feature extraction on the feature representation to obtain and output the three-dimensional coordinates of the contact position in space.
  • the specific three-dimensional coordinate system refer to the following to obtain the sample contact position Time coordinate system (refer to Figure 26).
  • the tactile measurement result includes the three-dimensional information of the contact force.
  • the feedforward neural network 200 includes a contact force estimation model and a contact force estimation model. Including: the second hidden layer and the second output layer, the method of measuring the three-dimensional information of the contact force is shown in Figure 10, including:
  • Step 501 Invoke the second hidden layer in the contact force estimation model to perform feature extraction on the difference features of the marking pattern to obtain a contact force feature representation.
  • the second hidden layer inputs the displacement of the marking point as the characteristic value to obtain the contact force characteristic representation, and the contact force characteristic representation As input to the second output layer.
  • Step 502 Invoke the second output layer in the contact force estimation model to process the contact force feature representation to obtain three-dimensional information of the contact force, the three-dimensional information including size and/or direction.
  • step 402 of using the feedforward neural network described above has been described in step 402 of using the feedforward neural network described above, and will not be repeated here.
  • the numbers of the second hidden layer and the second output layer in the above-mentioned contact force estimation model are both integers greater than zero, and the above-mentioned neurons can be selected according to the realization of different functions.
  • the number of the layer and the second output layer are 1 respectively, and the neurons are respectively selected based on Sigmoid hidden neurons and linear output neurons as examples, as shown in Figure 22.
  • the difference feature is input to the feedforward neural network 200, and the Sigmoid hidden neuron in the second hidden layer in the contact force estimation model is called to process the above feature values to obtain the feature representation of the contact position; the feature representation of the contact position will be As the input value is input to the second output layer, the linear output neuron in the second output layer will predict the feature representation and obtain the three-dimensional information of the contact force in space, that is, the magnitude and/or direction of the contact force and output .
  • size and/or direction includes: size only; or, direction only; or, size and direction.
  • the tactile measurement result includes the local radius of curvature of the contact surface
  • the feedforward neural network includes: a surface classification model and at least two curvature estimation models.
  • the at least two curvature estimation models include: a spherical curvature estimation model and a cylindrical curvature estimation model.
  • Step 601 Invoke the surface classification model to perform surface recognition on the displacement of the marking point, and obtain the surface type of the contact surface.
  • the surface classification model is a neural network model used to predict the surface type of the contacted object.
  • the surface type includes at least one of a spherical surface, a flat surface, and a cylindrical surface.
  • Step 602 Call the target curvature estimation model of the at least two curvature estimation models according to the surface type to predict the curvature of the contact surface, and obtain the local radius of curvature of the contact surface.
  • the chip calls the relevant curvature estimation model to estimate the curvature according to the type of the measured contact surface.
  • the surface classification model includes a third hidden layer and a third output layer, and step 601 includes the following sub-steps, as shown in FIG. 12, including:
  • Step 601a Invoke the third hidden layer in the surface classification model to perform surface recognition on the displacement of the marked point, and obtain a surface type feature representation.
  • Step 601b Invoke the third output layer in the surface classification model to process the surface type feature representation to obtain the surface type of the contact surface.
  • the marked point displacement is output as the surface type.
  • the surface type includes any one of plane, spherical or cylindrical surface.
  • the spherical estimation model includes a fourth hidden layer and a fourth output layer.
  • step 602 includes:
  • Step 602a calling the fourth hidden layer in the spherical surface estimation model to perform the first curvature prediction on the spherical surface to obtain the spherical curvature prediction feature representation.
  • Step 602b call the fourth output layer in the spherical surface estimation model to process the spherical surface curvature prediction feature representation to obtain the local radius of curvature of the spherical surface.
  • step 602 includes:
  • Step 6021 Invoke the fifth hidden layer in the cylindrical surface estimation model to perform a second curvature prediction on the cylindrical surface to obtain a characteristic representation of the cylindrical surface curvature prediction.
  • Step 6022 Call the fifth output layer in the cylindrical surface estimation model to process the cylindrical surface curvature prediction feature representation to obtain the local radius of curvature of the cylindrical surface.
  • the surface type of the contact surface can be, but is not limited to, a spherical surface, a cylindrical surface, or a flat surface. In some embodiments, the contact surface is a spherical surface as an example.
  • the hidden neurons and output neurons of the surface classification model can be It is set according to different realization functions, and the specific structure of the surface classification model of the present application is described in detail below (refer to FIG. 23).
  • the displacement of the marked point is input as the feature value to the above surface classification model.
  • the third hidden layer in the surface classification model performs surface recognition on the feature value to obtain the spherical type feature representation; the spherical type The feature representation is input to the third output layer as an input value, and the contact surface type is spherical; the chip calls the fourth hidden layer to predict the spherical curvature radius according to the contact surface type to be spherical, and obtains the spherical curvature prediction feature representation; the spherical curvature feature Indicates that the input is input to the fourth output layer, and the fourth output layer is called to process the spherical curvature prediction feature representation to obtain the local curvature radius of the spherical surface and output it.
  • the movement displacement of the marked point is input as a feature value to the above surface classification model, and the third hidden layer in the surface classification model performs surface recognition on the feature value to obtain a cylindrical surface type feature representation;
  • the characteristic representation of the cylindrical surface type is input as the input value to the third output layer, and the contact surface type is obtained as a cylindrical surface;
  • the chip uses the fifth hidden layer to predict the radius of curvature of the cylindrical surface according to the contact surface type to obtain the cylindrical surface curvature.
  • Predicted feature representation input the cylindrical curvature feature representation to the fifth output layer, call the fifth output layer to process the cylindrical curvature prediction feature representation, and obtain the local radius of curvature of the cylindrical surface and output it.
  • the above-mentioned local radius of curvature is continuous, not an intermittent interval.
  • the difference feature of the mark pattern is calculated according to the mark pattern in the adjacent images in the image sequence, as shown in FIG. 15, including:
  • Step 202a in the adjacent i-th frame image and i+1-th frame image in the image sequence, determine that the two closest marking patterns are the same marking pattern;
  • Step 202b Calculate the difference feature of the marking pattern according to the position (or position and deformation) of the marking pattern in the i-th frame image and the i+1-th frame image.
  • i is assumed to be 1, and in the adjacent first frame image and second frame image in the image sequence, the two closest marker points are determined to be the same marker point; The position in the 1 frame image and the second frame image, calculate the displacement of the marker point, as shown in Figure 15. Among them, the value of i is an integer.
  • FIG. 16 shows a flowchart of a method for training a feedforward neural network provided by an exemplary embodiment. According to an embodiment of the present application, the foregoing method is shown in FIG. 16, and includes:
  • Step 1601 Obtain training samples.
  • the training samples include a sample image sequence and sample tactile results.
  • the sample image sequence is an image sequence collected by an image sensing component in the tactile sensor.
  • Step 1602 Calculate the sample difference characteristics of the mark pattern according to the mark pattern in the adjacent images in the sample image sequence.
  • the sample difference feature of the marking pattern is calculated; or, according to the position and deformation (such as size) of the marking pattern in adjacent images in the sample image sequence ) To calculate the sample difference characteristics of the marking pattern.
  • the sample difference feature of the marking pattern includes: displacement of the marking point, or displacement and deformation of the marking point:
  • the sample difference characteristics of the marking pattern include: the displacement of the grid points in the grid, or the displacement of the grid points in the grid and the deformation of the grid lines.
  • step 1603 the feedforward neural network is called to process the sample difference characteristics of the mark pattern to obtain the predicted tactile result.
  • the number of hidden layers in the feedforward neural network is less than the threshold.
  • Step 1604 Perform error calculation on the predicted haptic result and the sample haptic result to obtain the error loss.
  • Step 1605 Train the feedforward neural network according to the error loss through the error back propagation algorithm to obtain the trained feedforward neural network.
  • the feedforward neural network used in this method is consistent with the above-mentioned neural network model.
  • the feedforward neural network is trained.
  • the specific structure of the feedforward neural network will not be repeated here.
  • the hidden layer and the output layer are provided in the feed-forward neural network; step 1603, that is, the step of calling the feed-forward neural network to process the sample difference features of the mark pattern to obtain the predicted tactile result , Including: calling the hidden layer in the feedforward neural network to perform feature extraction on the sample difference features of the mark pattern to obtain a feature representation; and calling the output layer in the feedforward neural network to process the feature representation to obtain a predicted tactile result.
  • the hidden layer is provided with n hidden neurons, where n is an integer; the hidden layer is constructed based on the logistic function Sigmoid hidden neurons; and the output layer is based on the normalized exponential function Softmax output Constructed by neurons or linear output neurons.
  • the feedforward neural network includes a position estimation model for estimating the contact position.
  • the position estimation model includes a first hidden layer and a first output layer.
  • the training method is shown in FIG. 17 and includes:
  • Step 1701 Obtain a first training sample.
  • the first training sample includes a first sample image sequence and sample contact positions.
  • the sample contact position is a position represented by coordinates in the form of three-dimensional coordinates.
  • Step 1702 Calculate the sample difference feature of the mark pattern according to the mark pattern in the adjacent images in the sample image sequence.
  • Step 1703 Call the position estimation model to process the sample difference characteristics of the marking pattern to obtain the predicted contact position.
  • Step 1704 Perform error calculation on the predicted contact position and the sample contact position to obtain the first error loss.
  • Step 1705 Train the position estimation model according to the first error loss through the Levenberg-Marquardt backpropagation algorithm to obtain the trained position estimation model.
  • the coordinates (x 1 , y 1 , z 1 ), the displacement of the marker points in the obtained image array is input to the first hidden layer and the first output layer, and the first output layer obtains the predicted coordinates of the sample contact position (x 1 ',y 1 ',z 1 '), the coordinates (x 1 ,y 1 ,z 1 ) and predicted coordinates (x 1 ',y 1 ',z 1 ) of the sample contact position are determined by the Levenberg-Marquardt backpropagation algorithm ') Perform processing to obtain the first error loss, train the position estimation model according to the first error loss, and obtain the trained position estimation model.
  • the feedforward neural network includes a contact force estimation model for estimating a three-dimensional contact force.
  • the contact force estimation model includes a second hidden layer and a second output layer.
  • the training method is shown in FIG. 18 and includes:
  • Step 1801 Obtain a second training sample.
  • the second training sample includes a second sample image sequence and sample three-dimensional information.
  • the sample three-dimensional information is calibrated based on data collected by a torque sensor arranged at the tail of the tactile sensor.
  • the three-dimensional information includes size And/or direction.
  • Step 1802 Calculate the sample difference feature of the mark pattern according to the mark pattern in the adjacent images in the sample image sequence.
  • Step 1803 Invoke the contact force estimation model to process the sample difference characteristics of the marking pattern to obtain predicted three-dimensional information.
  • Step 1804 Perform error calculation on the predicted 3D information and the sample 3D information to obtain the second error loss.
  • Step 1805 Train the contact force estimation model according to the second error loss through the Levenberg-Marquardt backpropagation algorithm to obtain the trained contact force estimation model.
  • the second training sample is the actual three-dimensional information of the displacement of the marker points in the image array and the sample contact force as an example, and the displacement of the marker points in the image array and the sample three-dimensional information (f x , f y , f z ), the displacement of the marker points in the acquired image array is input to the second hidden layer and the second output layer, and the second output layer obtains the predicted three-dimensional information of the sample contact force (f x ', f y ',f z '), the sample 3D information (f x , f y , f z ) and the predicted 3D information (f x ', f y ', f z ') are processed through the Levenberg-Marquardt backpropagation algorithm to obtain The second error loss is to train the position estimation model according to the second error loss to obtain the trained position estimation model.
  • the feedforward neural network includes a surface classification model for classifying contact surfaces, and the surface classification model includes a third hidden layer and a third output layer.
  • the training method is shown in FIG. 19 and includes:
  • Step 1901 Obtain a third training sample.
  • the third training sample includes a third sample image sequence and a sample surface type.
  • Step 1902 Calculate the sample difference feature of the mark pattern according to the mark pattern in the adjacent images in the sample image sequence.
  • Step 1903 Invoke the surface classification model to process the sample difference features of the marking pattern to obtain the predicted surface type.
  • Step 1904 Perform error calculation on the predicted surface type and the sample surface type to obtain a third error loss.
  • step 1905 the surface classification model is trained according to the third error loss through the scaled conjugate gradient back propagation algorithm to obtain the trained surface classification model.
  • the third training sample is the displacement of the marker points in the image array and the sample surface type as an example for illustration, the marker point displacement and the sample surface type (S 1 ) in the image array are acquired, and the acquired image The marker point displacement in the array is input to the third hidden layer and the third output layer.
  • the third output layer obtains the predicted surface type (S 1 ') of the contact surface, and the sample surface type (S 1 ) and the predicted surface type (S 1 ′) are processed to obtain the third error loss, and the surface classification model is trained according to the third error loss to obtain the trained surface classification model.
  • the structure of the above-mentioned surface classification model is shown in FIG. 23, and the third hidden layer can be provided with one or two.
  • the third hidden layer is provided with one layer as an example.
  • the surface classification model includes a third hidden layer and a third output layer.
  • the hidden layer is constructed based on Sigmoid hidden neurons.
  • the third output layer is constructed based on Softmax hidden neurons.
  • the Sigmoid hidden neurons are suitable for classifying objects.
  • And Softmax hides neurons to produce different output results corresponding to different shapes of the contact surface.
  • the curvature estimation model includes: a spherical estimation model.
  • the spherical estimation model includes a fourth hidden layer and a fourth output layer.
  • the training method is shown in FIG. 20 and includes:
  • Step 2010 Obtain a fourth training sample, where the fourth training sample includes the fourth sample image sequence and the local radius of curvature of the sample sphere;
  • Step 2020 Calculate the sample difference characteristics of the marker points according to the marker patterns in the adjacent images in the sample image sequence.
  • step 2030 the spherical estimation model is called to process the sample difference characteristics of the marking pattern to obtain the predicted radius of curvature.
  • Step 2040 Perform error calculation on the predicted radius of curvature and the local radius of curvature of the sample spherical surface to obtain a fourth error loss.
  • step 2050 the sphere estimation model is trained by the Levenberg-Marquardt back propagation algorithm according to the fourth error loss to obtain the trained sphere estimation model.
  • the displacement of the marker points in the image array and the local radius of curvature of the sample sphere input the displacement of the marker points in the acquired image array to the fourth hidden layer, and the fourth output layer obtains the predicted radius of curvature (R 1 ') of the sample spherical surface, which is reversed by Levenberg-Marquardt
  • the propagation algorithm processes the local radius of curvature (R 1 ) of the sample sphere and the predicted radius of curvature (R 1 ') of the sample sphere to obtain the fourth error loss.
  • the sphere estimation model is trained to obtain the trained Spherical estimation model.
  • the curvature estimation model includes: a cylindrical surface estimation model, and the cylindrical surface estimation model includes a fifth hidden layer and a fifth output layer.
  • the training method is shown in FIG. 21 and includes:
  • Step 2101 Obtain a fifth training sample, where the fifth training sample includes the fifth sample image sequence and the local radius of curvature of the sample cylindrical surface;
  • Step 2102 Calculate the difference characteristics of the marking patterns according to the marking patterns in the adjacent images in the sample image sequence.
  • Step 2103 Call the cylinder estimation model to process the difference features of the marking pattern to obtain the predicted radius of curvature.
  • Step 2104 Perform error calculation on the predicted radius of curvature and the local radius of curvature of the sample cylindrical surface to obtain the fifth error loss.
  • Step 2105 Train the cylinder estimation model according to the fifth error loss through the Levenberg-Marquardt backpropagation algorithm to obtain the trained cylinder estimation model.
  • the displacement of the marker points in the image array and the local radius of curvature of the sample cylindrical surface (R 2 ) is input to the fifth hidden layer, and the fifth output layer obtains the predicted radius of curvature (R 2 ') of the sample cylindrical surface, and passes the Levenberg-Marquardt Levenberg-Marquardt reaction
  • the direction propagation algorithm processes the local radius of curvature (R 2 ) of the sample cylindrical surface and the predicted radius of curvature (R 2 ') of the sample cylindrical surface to obtain the fifth error loss, and train the cylindrical surface estimation model according to the fifth error loss. Obtain the trained cylindrical estimation model.
  • the contacted surface is not limited to a spherical surface and a cylindrical surface.
  • the type of contact surface involved in the training sample and the number of use of each contact surface training sample are shown in Figure 24 and Figure 25, respectively.
  • This application selects contact surfaces with multiple shapes including cones, cylinders, triangular pyramids, triangular prisms, planes, etc., to train the surface classification model. Because a center with a contact force greater than 5.5N will cause a larger contact area Deformation, so for the tip surface (triangular pyramid in Figure 24), the application collects a contact force of less than 5.5N (as shown in the table in Figure 25).
  • the internal structure of the feedforward neural network 200 used when the feedforward neural network is not trained is the same, and the functions implemented by the internal structure are the same.
  • the structure of the feedforward neural network 200 used in training is described in detail, and the specific structure is shown in the above-mentioned feedforward neural network 200.
  • the tactile sensor system includes the above-mentioned tactile sensor and a chip 117 connected to the tactile sensor.
  • the chip 117 includes a programmable logic circuit and/or program instructions, and when the chip is running, it is used to execute the above-mentioned vision-based tactile measurement method.
  • the system is composed of the aforementioned tactile sensor and a chip 117.
  • the chip 117 is the same type of chip 117 as the chip in the structure of the vision-based tactile measurement device.
  • the chip 117 includes Programming logic circuits and/or program instructions, when the chip is running, is used to implement the above-mentioned vision-based tactile measurement method.
  • Fig. 26 shows a geometric model of the force/torque sensor measuring the contact point provided by an exemplary embodiment.
  • the actual contact position is obtained by the torque sensor 116 installed below the tactile sensor, and a virtual spatial coordinate system is established on the tactile sensor.
  • the tactile sensor measures the force f and the torque m generated by the contact force on the coordinate system established by the tactile sensor.
  • the force f is the contact force
  • the moment m can be written as follows:
  • r is the three-dimensional vector, that is, the position of the contact point relative to the coordinate system of the fingertip. It is a 3 ⁇ 3 antisymmetric matrix, used to represent the cross product. Because the matrix The rank of is two, so the solution of the formula can be written as follows:
  • the entire sphere is used to represent the sensing surface 111, therefore, there are two intersection points between the straight line and the sensing surface 111.
  • the inner product of the inner normal at one point and the force f measured by the tactile sensor is positive, that is, the actual contact position, and the position must be on the real sensing surface 111.
  • the other intersection point may or may not fall on the sensing surface 111, but the other intersection point can be omitted because the inner product of the other intersection point and the normal is negative.
  • the position estimation model is trained, and the training result is shown in FIG. 27.
  • the value of the correlation coefficient (R) of the position estimation model is close to 1, and the root mean square error (RMSE) is about 0.6mm, which means that the consistency of the trained position estimation model and the input data is highly related.
  • the contact force estimation model is trained, and the training result is shown in FIG. 28.
  • the performance results of the contact force estimation model in the dynamic force range of 1.5N to 8N are as follows: After training, the value of the correlation coefficient (R) is close to 1, and the root mean square error (RMSE) is about 0.25N. This means that the consistency of the trained contact force estimation model and the input data is highly correlated.
  • the surface classification model is trained, and the accuracy of the training result is shown in FIG. 29.
  • the spherical estimation model is trained, and the training result is shown in FIG. 30.
  • the training results of the spherical estimation model are as follows: After training, the value of the correlation coefficient (R) is about 0.9, and the root mean square error (RMSE) is about 8mm. This means that the trained spherical estimation model and the input The consistency of the data is highly relevant.
  • the cylindrical surface estimation model is trained, and the training result is shown in FIG. 31.
  • the training results of the spherical estimation model are as follows. After training, the correlation coefficient (R) is about 0.9, and the root mean square error (RMSE) is about 10mm. This means that the trained cylindrical estimation model is The consistency of the input data is highly relevant.
  • the flow chart of the method of using the tactile sensor system 300 is shown in FIG. 32.
  • the tactile sensor is provided with a sensing surface 111, and the inner surface of the sensing surface is provided with marking points 112, and
  • the image sensor assembly 115 facing the inner surface of the surface 111 collects the displacement of the marked point 112 on the image sequence formed by the inner surface of the flexible sensor 111 through the image sensor element 115, and inputs the displacement of the marked point 112 as a feature value to the above
  • the feedforward neural network measures the position of the contact point, the magnitude and/or direction of the three-dimensional contact force, and the local radius of curvature of the contact surface.
  • the feedforward neural network can also be trained by comparing the real value of the contact position measured by the torque sensor, the real value of the contact force, and the real value of the local curvature radius of the contact surface with the measured value measured by the tactile sensor.
  • the structural diagram of the vision-based tactile sensor is connected to the chip.
  • the vision-based tactile measurement device includes: a first acquisition module 311, a first calculation module 312, and a feedforward neural network 313, as shown in FIG. 33.
  • the first acquisition module 311 is configured to acquire an image sequence collected by the image sensing component 115 on the sensing surface, and the image of the image sequence includes a marking pattern.
  • the first calculation module 312 is configured to calculate the difference feature of the marking pattern according to the position of the marking pattern 112 in the adjacent images in the image sequence.
  • the feedforward neural network 313 is used to process the difference features of the marking pattern 112 to obtain the tactile measurement result.
  • the feedforward neural network is provided with a hidden layer and an output layer.
  • the hidden layer is used for feature extraction of the different features of the mark pattern to obtain a feature representation; the output layer is used for Feature represents processing to obtain tactile measurement results.
  • the hidden layer is constructed based on Sigmoid hidden neurons; the output layer is constructed based on Softmax output neurons or linear output neurons.
  • the above-mentioned tactile measurement result includes the contact position;
  • the feedforward neural network includes: a position estimation model, the position estimation model includes a first hidden layer and a first output layer, the first hidden layer is used for the description
  • the difference features of the marking pattern are feature extracted to obtain the contact position characteristic representation; the first output layer is used to process the contact position characteristic representation to obtain the contact position.
  • the above-mentioned tactile measurement results include three-dimensional information of contact force
  • the feedforward neural network includes a contact force estimation model
  • the contact force estimation model includes a second hidden layer and a second output layer.
  • the second output layer is used to process the characteristic expression of the contact force to obtain the three-dimensional information of the contact force.
  • the three-dimensional information includes the size and direction of the contact force. At least one.
  • the above-mentioned tactile measurement results include the local radius of curvature of the contact surface
  • the feedforward neural network includes: a surface classification model and at least two curvature estimation models; the surface classification model is used to detect differences in the marking patterns The feature performs surface recognition to obtain the surface type of the contact surface; the target curvature estimation model in at least two curvature estimation models is used to predict the curvature of the contact surface according to the surface type to obtain the local curvature radius of the contact surface.
  • the curvature estimation model includes: a spherical surface estimation model and a cylindrical surface estimation model.
  • the spherical surface estimation model is used to perform the first curvature prediction on the spherical surface when the surface type is a spherical surface to obtain the local curvature of the spherical surface Radius, when the surface type is a cylindrical surface, the cylindrical surface estimation model is used to predict the second curvature of the cylindrical surface to obtain the local radius of curvature of the cylindrical surface.
  • the surface classification model includes a third hidden layer and a third output layer.
  • the third hidden layer is used to perform surface recognition on the difference features of the marking pattern to obtain a surface type feature representation; the third output The layer is used to process the surface type feature representation to obtain the surface type of the contact surface.
  • the spherical surface estimation model includes a fourth hidden layer and a fourth output layer.
  • the fourth hidden layer is used to perform the first curvature prediction on the spherical surface to obtain the spherical curvature prediction feature representation; the fourth output layer , Used to process the prediction feature representation of spherical curvature to obtain the local radius of curvature of the spherical surface.
  • the cylindrical surface estimation model includes a fifth hidden layer and a fifth output layer.
  • the fifth hidden layer is used to perform a second curvature prediction on the cylindrical surface to obtain a cylindrical surface curvature prediction feature representation;
  • Five output layers used to process the prediction feature representation of the cylindrical surface curvature to obtain the local radius of curvature of the cylindrical surface.
  • the first calculation unit is configured to determine that the two closest marking patterns are the same marking pattern among the adjacent i-th frame image and the i+1-th frame image in the image sequence;
  • the second calculating unit is used to calculate the difference feature of the marking pattern according to the position (or position and deformation) of the marking pattern in the i-th frame image and the i+1-th frame image.
  • the marking pattern includes at least two marking points, and the sample difference feature of the marking pattern includes at least one of displacement and deformation of the marking point.
  • the marking pattern includes a grid
  • the difference feature of the marking pattern includes at least one of a displacement of a grid point in the grid and a deformation of a grid line.
  • Fig. 34 shows a schematic structural diagram of a training module of a vision-based tactile measurement device provided by an exemplary embodiment.
  • the training module of the vision-based tactile measurement device includes:
  • the second acquisition module 411 is used to acquire training samples.
  • the training samples include a sample image sequence and sample tactile results.
  • the sample image sequence is an image sequence collected by an image sensing component in the tactile sensor.
  • the tactile sensor includes a sensing surface and an image sensor. The component, the sensing surface is provided with a marking pattern, and the images of the image sequence include the marking pattern.
  • the second calculation module 412 is configured to calculate the sample difference characteristics of the marking pattern according to the marking patterns in the adjacent images in the sample image sequence.
  • the feedforward neural network model 413 is used to process the sample difference characteristics of the mark pattern to obtain the predicted tactile results, wherein the number of hidden layers in the feedforward neural network is less than the threshold.
  • the error calculation module 414 is configured to perform error calculation on the predicted haptic result and the sample haptic result to obtain the error loss.
  • the training module 415 is used to train the feedforward neural network model according to the error loss through the error back propagation algorithm to obtain the trained feedforward neural network model.
  • the feedforward neural network model is provided with a hidden layer and an output layer.
  • the hidden layer is used to extract the different features of the mark pattern to obtain a feature representation; the output layer is used to perform feature representation. Processing, get the tactile measurement result.
  • the hidden layer is constructed based on Sigmoid hidden neurons; the output layer is constructed based on Softmax output neurons or linear output neurons.
  • the feedforward neural network model includes a position estimation model for estimating the contact position
  • the second acquisition module 411 is used for acquiring a first training sample
  • the first training sample includes a first sample image sequence The contact position with the sample.
  • the estimation model is used to process the sample difference characteristics of the marking pattern to obtain the predicted contact position.
  • the error calculation module 414 is further configured to perform error calculation on the predicted contact position and the sample contact position to obtain a first error loss.
  • the training module 415 is also used to train the position estimation model according to the first error loss through the Levenberg-Marquardt backpropagation algorithm to obtain the trained position estimation model.
  • the feedforward neural network model includes a contact force estimation model for estimating the three-dimensional contact force
  • the second acquisition module 411 is also used to acquire a second training sample.
  • the second training sample includes a second sample.
  • the image sequence and the three-dimensional information of the sample contact force is calibrated based on the data collected by the torque sensor installed at the tail of the tactile sensor.
  • the three-dimensional information includes size and/or direction.
  • the contact force estimation model is used to process the sample difference characteristics of the marking pattern to obtain predicted three-dimensional information.
  • the error calculation module 414 is also used to perform error calculation on the predicted three-dimensional information and the sample three-dimensional information to obtain the second error loss.
  • the training module 415 is also used to train the contact force estimation model according to the second error loss through the Levenberg-Marquardt backpropagation algorithm to obtain the trained contact force estimation model.
  • the feedforward neural network model includes a surface classification model for classifying contact surfaces
  • the second acquisition module 411 is also used to acquire a third training sample
  • the third training sample includes a third sample image The type of sequence and sample contact surface.
  • the surface classification model is used to process the sample difference characteristics of the marking pattern to obtain the predicted surface type.
  • the error calculation module 414 is further configured to perform error calculation on the predicted surface type and the sample surface type to obtain a third error loss.
  • the training module 415 is also used to train the surface classification model according to the third error loss through the scaled conjugate gradient backpropagation algorithm to obtain the trained surface classification model.
  • the feedforward neural network model includes at least two curvature estimation models for estimating curvature
  • the second acquisition module 411 is used to acquire training samples
  • the training samples include the type of sample surface and sample tactile results
  • the second calculation module 412 is used to train the feedforward neural network according to the error loss through the error back propagation algorithm to obtain the trained feedforward neural network.
  • the curvature estimation model includes: a spherical estimation model, the second acquisition module 411 is used to acquire a fourth training sample, and the fourth training sample includes the type of the fourth sample surface and the local curvature of the sample sphere radius.
  • the spherical surface estimation model is used to process the sample difference characteristics of the marking pattern to obtain the predicted radius of curvature when the predicted surface type is spherical.
  • the error calculation module 414 is also used to perform error calculation on the predicted radius of curvature of the spherical surface and the local radius of curvature of the sample spherical surface to obtain the fourth error loss.
  • the training module 415 is also used to train the spherical surface estimation model according to the fourth error loss through the Levenberg-Marquardt backpropagation algorithm to obtain the trained spherical surface estimation model.
  • the curvature estimation model includes: a cylindrical surface estimation model, the second acquisition module 411 is used to acquire a fifth training sample, and the fifth training sample includes the type of the fifth sample surface and the sample cylindrical surface The local radius of curvature.
  • the cylindrical surface estimation model is used to call the cylindrical surface estimation model to process the difference characteristics of the marking pattern when the predicted surface type is a cylindrical surface to obtain the predicted radius of curvature.
  • the error calculation module 414 is also used to perform error calculation on the predicted radius of curvature of the cylinder and the local radius of curvature of the sample cylindrical surface to obtain the fifth error loss.
  • the training module 415 is also used to train the cylindrical surface estimation model according to the fifth error loss through the Levenberg-Marquardt backpropagation algorithm to obtain the trained cylindrical surface estimation model.
  • the embodiment of the present application also provides a computer device, which includes a memory and a processor, the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned feedforward neural network training method when the computer-readable instructions are executed.
  • Fig. 35 shows a schematic structural diagram of a computer device provided by an exemplary embodiment of the present application.
  • the computer device can be used to execute the above-mentioned feedforward neural network training method. Specifically:
  • the computer device 3500 includes a central processing unit (CPU) 3501, a system memory 3504 including a random access memory (RAM) 3502 and a read only memory (ROM) 3503, and a system bus 3505 connecting the system memory 3504 and the central processing unit 3501.
  • the computer equipment 3500 also includes a basic input/output system (I/O system) 3506 that helps transfer information between various devices in the computer, and a mass storage device for storing the operating system 3513, application programs 3514, and other program modules 3515 3507.
  • I/O system basic input/output system
  • the basic input/output system 3506 includes a display 3508 for displaying information and an input device 3509 such as a mouse and a keyboard for the user to input information.
  • the display 3508 and the input device 3509 are both connected to the central processing unit 3501 through the input and output controller 3510 connected to the system bus 3505.
  • the basic input/output system 3506 may also include an input and output controller 3510 for receiving and processing input from multiple other devices such as a keyboard, a mouse, or an electronic stylus.
  • the input and output controller 3510 also provides output to a display screen, a printer, or other types of output devices.
  • the mass storage device 3507 is connected to the central processing unit 3501 through a mass storage controller (not shown) connected to the system bus 3505.
  • the mass storage device 3507 and its associated computer-readable medium provide non-volatile storage for the computer device 3500. That is, the mass storage device 3507 may include a computer-readable medium (not shown) such as a hard disk or a CD-ROM drive.
  • Computer-readable media may include computer storage media and communication media.
  • Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storing information such as computer readable instructions, data structures, program modules or other data.
  • Computer storage media include RAM, ROM, EPROM, EEPROM, flash memory or other solid-state storage technologies, CD-ROM, DVD or other optical storage, tape cartridges, magnetic tape, disk storage or other magnetic storage devices.
  • RAM random access memory
  • ROM read-only memory
  • EPROM Erasable programmable read-only memory
  • EEPROM electrically erasable programmable read-only memory
  • the computer device 3500 may also be connected to a remote computer on the network through a network such as the Internet to operate. That is, the computer device 3500 can be connected to the network 3512 through the network interface unit 3511 connected to the system bus 3505, or in other words, the network interface unit 3511 can also be used to connect to other types of networks or remote computer systems (not shown).
  • the aforementioned memory also includes one or more programs, and one or more programs are stored in the memory and configured to be executed by the CPU.
  • the embodiment of the present application also provides a computer-readable storage medium storing computer-readable instructions, wherein the computer-readable instructions are characterized in that, when the computer-readable instructions are executed by a processor, the above vision-based tactile measurement method is implemented, or the above feedforward Training method of neural network.
  • the embodiment of the present application also provides a robot system, the robot system includes: a chip and a tactile sensor, the tactile sensor is provided in at least one of the fingertip part and the skin part, the tactile sensor includes a flexible sensing surface and an orientation flexible sensor.
  • the image sensing component is arranged on the inner surface of the sensing surface. Marking points are arranged in the flexible sensing surface.
  • the image sensing component is connected to the chip.
  • the chip includes at least one of a programmable logic circuit and program instructions. , Used to implement the above-mentioned vision-based tactile measurement method.
  • the embodiment of the present application also provides a flow chart of a method for using the touch sensor system 300, as shown in FIG. 32.
  • the method of use is suitable for the above-mentioned touch sensor, the feedforward neural network in the touch sensor, and the above-mentioned feedforward neural network.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Human Computer Interaction (AREA)
  • Robotics (AREA)
  • Mechanical Engineering (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

一种基于视觉的触觉测量方法，该方法由芯片执行，芯片与触觉传感器相连，触觉传感器包括传感面和图像传感组件，传感面设置有标记图案，该方法包括：获取图像传感组件对传感面采集的图像序列，图像序列的图像中包括标记图案；根据图像序列中的相邻图像内的标记图案，计算标记图案的差异特征；及调用前馈神经网络对标记图案的差异特征进行处理，得到触觉测量结果。

Description

基于视觉的触觉测量方法、装置、芯片及存储介质
本申请要求于2019年05月16日提交中国专利局，申请号为201910411693.6、发明名称为“基于视觉的触觉测量方法、装置、设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及人机交互领域，特别涉及一种基于视觉的触觉测量方法、装置、芯片及存储介质。
背景技术
触觉传感器是用于模仿触觉功能的传感器，可以对接触物体进行触觉测量，比如接触位置、接触力等。目前，触觉传感器多应用于机器人领域。
相关技术中提供了一种触觉传感器，该触觉传感器设置有半圆形的柔性传感面，该柔性传感面的内表面设置有阵列排布的多个标记点，以及朝向内表面设置的图像传感组件。在柔性传感面的外表面与物体接触后，柔性传感面会发生形变，导致内表面的多个标记点因形变而改变位置。图像传感组件采集柔性传感面的内表面图像，将内表面图像传输至芯片。芯片内设置有卷积神经网络(CNN)，通过卷积神经网络对内表面图像进行处理，得到接触力的分析结果。
上述卷积神经网络的训练过程较为复杂，需要多达两万个训练样本才能取得较好的训练效果。
发明内容
根据本申请的各种实施例提供了一种基于视觉的触觉测量方法、装置、芯片、及存储介质，一种前馈神经网络的训练方法、装置、计算机设备和存 储介质，一种触觉传感器系统，以及一种机器人系统。
一种基于视觉的触觉测量方法，所述方法由芯片执行，所述芯片与触觉传感器相连，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案；
所述方法包括：获取所述图像传感组件对所述传感面采集的图像序列，所述图像序列的图像中包括所述标记图案；根据所述图像序列中的相邻图像内的所述标记图案，计算所述标记图案的差异特征；及调用前馈神经网络对所述标记图案的差异特征进行处理，得到触觉测量结果；其中，所述前馈神经网络内的隐藏层数量小于阈值。
一种前馈神经网络的训练方法，由计算机设备执行，所述方法包括：
获取训练样本，所述训练样本包括样本图像序列和样本触觉结果，所述样本图像序列是触觉传感器内的图像传感组件采集的图像序列，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案，所述图像序列的图像中包括所述标记图案；根据所述样本图像序列中的相邻图像内的所述标记图案的位置，计算所述标记图案的样本差异特征；调用前馈神经网络对所述标记图案的样本差异特征进行处理，得到预测触觉结果；所述前馈神经网络中的隐藏层数量少于阈值；对所述预测触觉结果和所述样本触觉结果进行误差计算，得到误差损失；及通过误差反向传播算法根据所述误差损失对所述前馈神经网络进行训练，得到训练后的前馈神经网络。
一种基于视觉的触觉测量装置，所述装置应用于芯片中，所述芯片与触觉传感器相连，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案；
所述装置包括：第一获取模块，用于获取所述图像传感组件对所述传感面采集的图像序列，所述图像序列的图像中包括所述标记图案；第一计算模块，用于根据所述图像序列中的相邻图像内的所述标记图案，计算所述标记图案的差异特征；及前馈神经网络，用于对所述标记图案的差异特征进行处理，得到触觉测量结果；其中，所述前馈神经网络内的隐藏层数量小于阈值。
一种前馈神经网络的训练装置，所述装置包括：
第二获取模块，用于获取训练样本，所述训练样本包括样本图像序列和样本触觉结果，所述样本图像序列是触觉传感器内的图像传感组件采集的图像序列，所述触觉传感器包括柔性传感面和朝向所述柔性传感面的内表面设置的图像传感组件，所述柔性传感面内设置有标记图案，所述图像序列的图像中包括所述标记图案的位置；第二计算模块，用于根据所述样本图像序列中的相邻图像内的所述标记图案的位置，计算所述标记图案的样本差异特征；前馈神经网络模型，用于对所述标记图案的样本差异特征进行处理，得到预测触觉结果，所述前馈神经网络中的隐藏层数量少于阈值；误差计算模块，用于对所述预测触觉结果和所述样本触觉结果进行误差计算，得到误差损失；及训练模块，用于通过误差反向传播算法根据所述误差损失对所述前馈神经网络进行训练，得到训练后的前馈神经网络。
一种芯片，所述芯片包括可编程逻辑电路和程序指令中的至少一种，当所述芯片运行时所述芯片用于执行如上方面所述的基于视觉的触觉测量方法。
一种触觉传感器系统，所述系统包括：触觉传感器和芯片，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案，所述图像传感组件与所述芯片相连；所述芯片包括可编程逻辑电路和程序指令中的至少一种，当所述芯片运行时所述芯片用于执行如上方面所述的基于视觉的触觉测量方法。
一种计算机设备，包括存储器和处理器，所述存储器存储有计算机可读指令，所述处理器执行所述计算机可读指令时实现上述前馈神经网络的训练方法的步骤。
一种计算机可读存储介质，存储有计算机可读指令，所述计算机可读指令被处理器执行时实现上述的基于视觉的触觉测量方法的步骤，或者，上述的前馈神经网络的训练方法的步骤。
一种机器人系统，所述机器人系统包括：芯片和触觉传感器，所述触觉 传感器设置在指尖部位和皮肤部位中的至少一个部位，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案，所述图像传感组件与所述芯片相连；所述芯片包括可编程逻辑电路和程序指令中的至少一种，当所述芯片运行时所述芯片用于执行如上方面所述的基于视觉的触觉测量方法。
本申请的一个或多个实施例的细节在下面的附图和描述中提出。本申请的其它特征、目的和优点将从说明书、附图以及权利要求书变得明显。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一个示例性实施例提供的相关技术中的触觉传感器的结构示意图；
图2是本申请一个示例性实施例提供的触觉传感器系统的结构示意图；
图3是本申请一个示例性实施例提供的柔性传感面的示意图；
图4是本申请一个示例性实施例提供的触觉传感器的使用方法流程图；
图5是是本申请一个示例性实施例提供的标记点的位移记录示意图；
图6是本申请一个示例性实施例提供的前馈神经网络的结构示意图；
图7是本申请一个示例性实施例提供的前馈神经网络的的使用方法流程；
图8是本申请一个示例性实施例提供前馈神经网络中的各模型结构示意图；
图9是本申请一个示例性实施例提供的位置估计模型的使用方法流程图；
图10是本申请另一个示例性实施例提供的接触力估计模型的使用方法 流程图；
图11是本申请一个示例性实施例提供的表面分类模型和曲率估计模型的使用方法流程图；
图12是本申请一个示例性实施例提供的表面分类模型的使用方法的流程图；
图13是本申请一个示例性实施例提供的球面的曲率估计模型的使用方法的流程图；
图14是本申请一个示例性实施例提供的圆柱面的曲率估计模型的使用方法流程图；
图15是本申请一个示例性实施例提供的图像阵列上标记点位移的计算方法的流程图；
图16是本申请一个示例性实施例提供的基于视觉的触觉测量的训练方法的流程图；
图17是本申请一个示例性实施例提供的位置估计模型的训练方法的流程图；
图18是本申请一个示例性实施例提供的接触力估计模型的训练方法流程图；
图19是本申请一个示例性实施例提供的表示表面分类模型的训练方法流程图；
图20是本申请一个示例性实施例提供的球面估计模型的训练方法流程图；
图21是本申请另一个示例性实施例提供的圆柱面估计模型的训练方法流程图；
图22是本申请一个示例性实施例提供的除表面分类模型之外的其它模型的结构示意图；
图23是本申请一个示例性实施例提供的表面分类模型的模型结构示意图；
图24是本申请一个示例性实施例提供的用于训练前馈神经网络的样本曲面示意图；
图25是本申请一个示例性实施例提供的训练样本曲面的使用个数的表格；
图26是本申请一个示例性实施例提供的触觉传感器的几何模型示意图；
图27是本申请一个示例性实施例提供的位置估计模型的训练结果示意图；
图28是本申请一个示例性实施例提供的接触力估计模型的训练结果示意图；
图29是本申请一个示例性实施例提供的表示表面分类模型的训练结果正确率的混淆矩阵；
图30是本申请一个示例性实施例提供的球面估计模型的训练结果示意图；
图31是本申请一个示例性实施例提供的圆柱面估计模型的训练结果示意图；
图32是本申请一个示例性实施例提供的触觉传感器系统的使用方法的流程图。
图33是本申请一个示例性实施例提供的基于视觉的触觉测量装置的框图；
图34是本申请另一个示例性实施例提供的前馈神经网络的训练装置的框图；
图35是本申请一个示例性实施例提供的计算机设备的框图。
具体实施方式
下面将参照附图更详细地描述本申请的实施例。虽然附图中显示了本申请的某些实施例，然而应当理解的是，本申请可以通过各种形式来实现，而且不应该被解释为限于这里阐述的实施例，相反提供这些实施例是为了更加 透彻和完整地理解本申请。应当理解的是，本申请的附图及实施例仅用于示例性作用，并非用于限制本申请的保护范围。
本文使用的术语“包括”及其变形是开放性包括，即“包括但不限于”。术语“基于”是“至少部分地基于”。术语“一个实施例”表示“至少一个实施例”；术语“另一实施例”表示“至少一个另外的实施例”。其他术语的相关定义将在下文描述中给出。
首先，对本申请实施例提供的若干个名词进行简介：
前馈神经网络(Feedforward Neural Network)模型：是一种具有单向结构的人工神经网络。前馈神经网络包括至少两个神经网络层。其中，每一个神经网络层包含若干个神经元，各个神经元分层排列，同一层的神经元之间没有互相连接，层间信息的传送只沿一个方向进行。
逻辑斯蒂(Sigmoid)函数：是一种呈“S”型的函数，用来描述增长趋势为起初阶段大致是指数增长；然后随着开始变得饱和，增加变慢；最后，达到成熟时增加停止的过程。
归一化指数(Softmax)函数：是一种可以将一个含任意实数的向量“压缩”到另一个实向量中，使得每一个元素的范围都在(0,1)之间，并且所有元素的和为1的函数。
隐藏层：是一种用于输入或者分析数据的神经网络层结构。
输出层：是一种用于输出结果的神经网络层结构。
在相关技术中提供了一种触觉传感器，如图1所示。该触觉传感器包括：硅胶传感面11、设置在硅胶传感面11内表面上的标记点12，3D打印技术制成的触觉传感器的模型前端13，该模型前端13用于固定硅胶传感面11，透镜14、图像传感组件15，用于固定图像传感组件的触觉传感器模型尾端16和发光二极管(LED)组成的圆环17。该技术采用基于贝叶斯定理(Bayesian)的概率模型来区分接触点的位置、接触棱边的曲率半径以及朝向，并且在最新的研究中使用了卷积神经网络模型(CNN)的算法来实现以上测试结果， 但由于基于贝叶斯定理(Bayesian)的概率模型不是连续的估测算法，因此，该技术只能将所接触物体的曲率半径进行不连续的区分，如只能区分20mm或30mm，同时，采用卷积神经网络模型(CNN)的算法存在训练样本数量大的问题，比如，需要两万张以上的样本图像才能训练到较好的模型效果。
在本申请提供的一些实施例中，利用前馈神经网络提供了一种基于视觉的触觉测量方案。该方法利用图像阵列中连续两幅图像上的标记点的位移作为特征值输入至前馈神经网络，得到接触点的位置、接触力的大小和/或方向，接触表面的曲率半径等信息，相比于相关技术中所采用的卷积神经网络模型(CNN)，该方法将输入的特征值(只需要标记点的位移和/或形变，而不需要整张图像)进行简化，同时，训练样本大幅度减少，提高了神经网络的训练效率，从而满足将使用触觉传感器的方法简化，并且在不需要大量样本的训练下，该前馈神经网络可以达到同样的效果(或更优的效果)的需求。
在本申请提供的一些实施例中，该基于视觉的触觉测量方法应用于芯片117中，该芯片117可以是CPU、GPU、神经网络芯片、微处理器、或FPGA电路中的任意一种，此处不作限定，该芯片117与触觉传感器相连，触觉传感器包括传感面111和图像传感组件115，传感面111设置有标记图案112。可选地，传感面111是柔性传感面，能够在与其它物体接触时发生形变。图像传感组件115可以是摄像头。该摄像头可朝向传感面111的内表面来设置。
图2示出了本申请的一个示例性实施例提供的一种触觉传感器系统300的结构示意图。该触觉传感器包括传感面111、底座113、立柱114和朝向传感面111的内表面设置的图像传感组件115构成，底板116用于放置该触觉传感器。
图3示出了本申请的一个示例性实施例提供的一种传感面111的内表面示意图，该传感面111的内表面上设置有标记图案112。
在一些实施例中，该传感面111的形状不作限定，该传感面111可以是矩形、六边形、圆形、椭圆形、半球形、或平面等形状中的任意一种，在本 实施例的一个示例中，选用半球形的柔性传感面111为例进行说明，如图3所示。
在一些实施例中，标记图案采用至少两个标记点来实现，或者，采用网格来实现，或者，采用标记点和网格来实现。网格是具有交叉的网格线的图案，相交的网格线上形成网格点。
以标记图案采用标记点来实现为例，在该传感面111的内表面上(或内部)设置有标记点112，标记点112可以形成阵列排布或者非阵列排布，如，形成4×4或6×6的矩形阵列，或者，形成圆形的非阵列排布。该阵列在传感面111的内表面上至少存在一个，如，该传感面111上设置有两个阵列，阵列的个数能反映更多的形变，计算量也会随着增加。
相邻标记点112之间的距离可以是相等的，也可以是不相等的，在相邻标记点112之间的距离相等的时候，该标记点112的位移变化均匀。标记点112可以居中排布在传感面111的内表面上，如，在传感面111上设置有4×4的矩形阵列的标记点112，也可以沿着传感面111的边缘排布。该标记点的颜色可以是任意颜色，本申请中选用黑色标记点112，与白色的传感面111加以区分，能更好地表示标记点112的位移。
在一些实施例中，黑色标记点112阵列居中排布在传感面111的内表面上，并且距标记点阵列中位于边缘的标记点112应与传感面111上的各边缘距离相等，相邻标记点112之间的距离相等。作为另一个示例，选用圆形传感面111，标记点112的个数为6×6的矩形阵列为例进行说明，如图3所示。
本申请实施例对标记图案112的具体实现形式不加以限定。
图4示出了本申请一个示例性实施例提供的基于视觉的触觉测量方法的流程图，该方法可以由图1中的芯片117来执行，该方法包括：
步骤201，获取图像传感组件对内表面采集的图像序列，图像序列的图像中包括标记图案。
被接触物体与接触传感器的传感面111接触时，传感面111发生形变，图像传感组件以一定的频率连续地拍摄传感面111的内表面并向芯片传输图 像，因此形成图像阵列。
以标记图案包括标记点为例，上述频率可以根据标记点112的位移来设定，如30帧/秒或60帧/秒。此外，图像传感组件115在高频率的拍摄状态下，触觉传感器还可以检测被接触物体的滑移，甚至可以检测到较大的突发性力。
步骤202，根据图像序列中的相邻图像内的标记图像，计算标记图像的差异特征。
在一些实施例中，图像阵列中的相邻的图像是相邻的两张。
当标记图案包括标记点时，标记图案的差异特征包括标记点的位移和形变中的至少一种，比如包括标记点的位移，还比如包括标记点的位移和形变。
当标记图案包括网格时，标记图案的差异特征包括网格中网格点的位移和网格线的形变中的至少一种，比如包括网格中网格点的位移，还比如包括网格中的网格点的位移和网格线的形变。
以标记图案包括标记点为例，上述芯片根据两个相邻图像中的最接近的两个标记点112标记为同一标记，就能够跟踪每一个标记点的运动，即计算出标记点112的位移作为差异特征，如图5所示。
步骤203，调用前馈神经网络对标记图案的差异特征进行处理，得到触觉测量结果。
前馈神经网络可以是一个或多个神经网络模型，每个神经网络模型对应不同的功能。前馈神经网络中的隐藏层的层数小于阈值。在一些实施例中，该阈值为2。
调用前馈神经网络中各个模型的隐藏层和输出层，根据前馈神经网络中的实现不同功能的模型对标记图案的差异特征进行处理，得到触觉测量结果。
在一些实施例中，可以根据测量需要设计不同功能的神经网络模型、隐藏层和输出层的个数，本申请的实施例中以隐藏层和输出层的个数分别为1进行说明。
综上所述，本申请实施例提供的方法，通过采用标记图案的差异特征作为输入特征，相比于相关技术中采用图像作为输入特征，能够减少输入特征 的数量，从而减少计算量；同时，通过采用隐藏层的层数小于阈值的前馈神经网络来进行特征提取和预测，相比于层数较多的CNN网络，能够使用较少的计算量来预测出相近或更优的触觉测量结果。
图6示出了一个示例性实施例提供的一种前馈神经网络200的结构示意图。该前馈神经网络200内设置有隐藏层202和输出层203。
在基于图4的可选实施例中，上述前馈神经网络200对标记图案112的差异特征进行处理得到测量结果的方法包括如下步骤，如图7所示：
步骤301，调用前馈神经网络200中的隐藏层201对标记图案112的差异特征进行特征提取，得到特征表示。
步骤302，调用前馈神经网络中的输出层202对特征表示进行处理，得到触觉测量结果。
上述隐藏层201内设置有n个隐藏神经元，n为整数。作为本实施例的一个示例，该隐藏层201是基于Sigmoid隐藏神经元构建的，上述输出层202是基于Softmax输出神经元或线性输出神经元构建的。
根据本申请的实施例，隐藏层的神经元数量可以是任意大于零的整数，被输入的特征值的个数也可以是任意大于零的整数，上述神经元可以根据实现功能的不同来进行设计。
在一些实施例中，标记图案112采用36个标记点来实现，以隐藏层201内设置有100个隐藏神经元为例进行说明，向该前馈神经网络200中的输入层201输入72个特征值，该特征值为36个标记点112的位移(x1，y1，x2，y2，…，x36，y36)。
综上所述，在触觉传感器结构相似的前提下，本申请所涉及的触觉传感器结构较为简单，同时，采用将标记点在连续的图像阵列上的位移作为特征值输入至神经网络模型的方法，及设计含有简单的隐藏层的前馈神经网络对被接触的物体进行测量，简化了测量过程，减少了训练样本的数量。
前馈神经网络的结构中的隐藏层以单向的方式传递信息，该隐藏层至少含有一个，参考图6示出的一个示例性实施例提供的一种前馈神经网络200的示意图。
上述触觉测量结果，包括如下三种特征中的一种：
第一，接触位置；
第二，接触力的三维信息，该三维信息包括接触力的大小和/或方向；
第三，接触表面的局部曲率半径。
上述前馈神经网络包括：用于估计接触点位置的位置估计模型、用于估计接触力三维信息(接触力的大小和/或方向)的接触力估计模型、用于对接触表面进行分类的表面分类模型、用于对接触表面的局部曲率半径进行估计的曲率估计模型，如图8所示。
下面用图9来对接触位置的测量过程进行说明，根据本申请的实施例，触觉测量结果包括接触位置，前馈神经网络包括：位置估计模型，位置估计模型包括第一隐藏层和第一输出层，接触位置的测量方法如图9所示，包括：
步骤401，调用位置估计模型中的第一隐藏层对标记图案的差异特征进行特征提取，得到接触位置特征表示；
以标记图案包括阵列排布的标记点，标记图案的差异特征包括标记点的位移为例，上述第一隐藏层用于对输入的标记点的位移进行特征提取，该第一隐藏层是基于Sigmoid隐藏神经元构建的，上述接触位置特征表示是采用向量形式对于接触位置对应的特征表示进行的表示。在上述描述前馈神经网络的使用步骤301中已说明，此处不再赘述。
步骤402，调用位置估计模型中的第一输出层对接触位置特征表示进行处理，得到接触位置。
在上述描述前馈神经网络的使用步骤302中已说明，此处不再赘述。
在一些实施例中，上述位置估计模型中隐藏层和输出层的个数均为大于零的整数，上述神经元可以根据实现不同的功能来进行选择，本申请以第一 隐藏层和第一输出层的个数分别为1个，神经元分别选用基于Sigmoid隐藏神经元和线性输出神经元为例说明，如图22所示。
将特征值输入至该前馈神经网络200，调用位置估计模型中的第一隐藏层中的Sigmoid隐藏神经元的对上述特征值进行处理，得到接触位置特征表示；该接触位置的特征表示将作为输入值输入至第一输出层，第一输出层中的线性输出神经元将对特征表示进行特征提取，得到接触位置在空间上的三维坐标并输出，具体的三维坐标系参照下文获取样本接触位置时的坐标系(参考图26)。
下面用图10来对接触力的三维信息的测量过程进行说明，根据本申请的实施例，触觉测量结果包括接触力的三维信息，上述前馈神经网络200包括接触力估计模型，接触力估计模型包括：第二隐藏层和第二输出层，测量接触力的三维信息的方法如图10所示，包括：
步骤501，调用接触力估计模型中的第二隐藏层对标记图案的差异特征进行特征提取，得到接触力特征表示。
以标记图案包括阵列排布的标记点，标记图案的差异特征包括标记点的位移为例，上述第二隐藏层将标记点的位移作为特征值输入，得到接触力特征表示，将接触力特征表示作为第二输出层的输入。
步骤502，调用接触力估计模型中的第二输出层对接触力特征表示进行处理，得到接触力的三维信息，三维信息包括大小和/或方向。
在上述描述前馈神经网络的使用步骤402中已说明，此处不再赘述。
在一些实施例中，上述接触力估计模型中第二隐藏层和第二输出层的个数均为大于零的整数，上述神经元可以根据实现不同的功能来进行选择，本申请以第二隐藏层和第二输出层的个数分别为1个，神经元分别选用基于Sigmoid隐藏神经元和线性输出神经元为例说明，如图22所示。将差异特征输入至该前馈神经网络200，调用接触力估计模型中的第二隐藏层中的Sigmoid隐藏神经元的对上述特征值进行处理，得到接触位置特征表示；该接 触位置的特征表示将作为输入值输入至第二输出层，第二输出层中的线性输出神经元将对特征表示进行预测，得到接触力在空间上的三维信息，也即得到接触力的大小和/或方向并输出。
其中，“大小和/或方向”包括：仅大小；或，仅方向；或，大小和方向。
下面用图11来对局部曲率半径的测量过程进行说明，根据本申请的实施例，触觉测量结果包括接触表面的局部曲率半径，前馈神经网络包括：表面分类模型和至少两个曲率估计模型。作为本实施例的一个示例，至少两个曲率估计模型包括：球面曲率估计模型和柱面曲率估计模型。
测量接触表面的局部曲率半径的方法如图11所示，包括：
步骤601，调用表面分类模型对标记点的位移进行表面识别，得到接触表面的表面类型。
表面分类模型是用于对被接触的物体的表面类型进行预测的神经网络模型，表面类型包括球面、平面和圆柱面中的至少一种。
步骤602，根据表面类型调用至少两个曲率估计模型中的目标曲率估计模型对接触表面进行曲率预测，得到接触表面的局部曲率半径。
芯片根据测量的接触表面的类型来调用相关的曲率估计模型进行曲率估计。
在基于上述图11的可选实施例中，表面分类模型包括第三隐藏层和第三输出层，步骤601包括如下子步骤，如图12所示，包括：
步骤601a，调用表面分类模型中的第三隐藏层对标记点的位移进行表面识别，得到表面类型特征表示。
步骤601b，调用表面分类模型中的第三输出层对表面类型特征表示进行处理，得到接触表面的表面类型。
通过调用表面分类模型，将标记点位移输出为表面类型。表面类型包括：平面、球面或圆柱面中的任意一种。
当表面类型为球面时，进入图13所示的步骤；当表面类型为圆柱面时，进入图14所示的步骤。
在基于上述图11的可选实施例中，球面估计模型包括第四隐藏层和第四输出层，步骤602如图13所示，包括：
步骤602a，调用球面估计模型中的第四隐藏层对球面进行第一曲率预测，得到球面曲率预测特征表示。
步骤602b，调用球面估计模型中的第四输出层对球面曲率预测特征表示进行处理，得到球面的局部曲率半径。
在基于上述图11的可选实施例中，圆柱面估计模型包括第五隐藏层和第五输出层，步骤602如图14所示，包括：
步骤6021，调用圆柱面估计模型中的第五隐藏层对圆柱面进行第二曲率预测，得到圆柱面曲率预测特征表示。
步骤6022，调用圆柱面估计模型中的第五输出层对圆柱面曲率预测特征表示进行处理，得到圆柱面的局部曲率半径。
该接触表面的表面类型可以是但不局限于球面、圆柱面、或平面等，在一些实施例中，以接触表面为球面为例进行说明，该表面分类模型的隐藏神经元和输出神经元可以根据实现功能的不同来设置，本申请的表面分类模型的具体结构在下文中详细说明(参考图23)。
当被检测的接触面为球面时，将标记点的位移作为特征值输入至上述表面分类模型，表面分类模型中的第三隐藏层对特征值进行表面识别，得到球面类型特征表示；球面类型的特征表示作为输入值输入至第三输出层，得到接触表面类型为球面；芯片根据接触表面类型为球面，调用第四隐藏层对球面曲率半径进行预测，得到球面曲率预测特征表示；将球面曲率特征表示输入至第四输出层，调用第四输出层对球面曲率预测特征表示进行处理，得到球面的局部曲率半径并输出。
当被检测的接触面为圆柱面时，将标记点的移动位移作为特征值输入至上述表面分类模型，表面分类模型中的第三隐藏层对特征值进行表面识别， 得到圆柱面类型特征表示；圆柱面类型的特征表示作为输入值输入至第三输出层，得到接触表面类型为圆柱面；芯片根据接触表面类型为圆柱面，调用第五隐藏层对圆柱面曲率半径进行预测，得到圆柱面曲率预测特征表示；将圆柱面曲率特征表示输入至第五输出层，调用第五输出层对圆柱面曲率预测特征表示进行处理，得到圆柱面的局部曲率半径并输出。
其中，上述局部曲率半径是连续的，而不是断续的区间。
在基于上述各个实施例的可选实施例中，根据图像序列中的相邻图像内的标记图案，计算标记图案的差异特征，如图15所示，包括：
步骤202a，在图像序列中相邻的第i帧图像和第i+1帧图像中，确定最接近的两个标记图案为相同的标记图案；
步骤202b，根据标记图案在第i帧图像和第i+1帧图像中的位置(或，位置和形变)，计算标记图案的差异特征。
在一些实施例中，以i为1进行说明，在图像序列中相邻的第1帧图像和第2帧图像中，确定最接近的两个标记点为相同的标记点；根据标记点在第1帧图像和第2帧图像中的位置，计算标记点的位移，如图15所示。其中，i的取值为整数。
下面对上述各个实施例中提及的前馈神经网络的训练方法进行阐述。
图16示出了一个示例性实施例提供的前馈神经网络的训练方法的流程图。根据本申请的实施例，上述方法如图16所示，包括：
步骤1601，获取训练样本，训练样本包括样本图像序列和样本触觉结果，样本图像序列是触觉传感器内的图像传感组件采集的图像序列。
步骤1602，根据样本图像序列中的相邻图像内的标记图案，计算标记图案的样本差异特征。
示意性的，根据样本图像序列中的相邻图像内的标记图案的位置，计算标记图案的样本差异特征；或者，根据样本图像序列中的相邻图像内的标记 图案的位置和形变(比如大小)，计算标记图案的样本差异特征。
当标记图案包括至少两个标记点，标记图案的样本差异特征包括：标记点的位移，或，所述标记点的位移和形变：
当标记图案包括网格，标记图案的样本差异特征包括：网格中网格点的位移，或，网格中的网格点的位移和网格线的形变。
步骤1603，调用前馈神经网络对标记图案的样本差异特征进行处理，得到预测触觉结果。其中，前馈神经网络中的隐藏层数量少于阈值。
步骤1604，对预测触觉结果和样本触觉结果进行误差计算，得到误差损失。
步骤1605，通过误差反向传播算法根据误差损失对前馈神经网络进行训练，得到训练后的前馈神经网络。
该方法所用的前馈神经网络与上述神经网络模型一致，此处是对该前馈神经网络进行训练，该前馈神经网络的具体结构，此处不再赘述。
在一些实施例中，该前馈神经网络内设置有所述隐藏层和输出层；步骤1603，也就是调用前馈神经网络对所述标记图案的样本差异特征进行处理，得到预测触觉结果的步骤，包括：调用前馈神经网络中的隐藏层对标记图案的样本差异特征进行特征提取，得到特征表示；及调用前馈神经网络中的输出层对所述特征表示进行处理，得到预测触觉结果。
在一些实施例中，该隐藏层内设置有n个隐藏神经元，n为整数；隐藏层是基于逻辑斯蒂函数Sigmoid隐藏神经元构建的；及该输出层是基于归一化指数函数Softmax输出神经元或线性输出神经元构建的。
根据本申请的实施例，前馈神经网络包括用于估计接触位置的位置估计模型，位置估计模型包括第一隐藏层和第一输出层，该训练方法如图17所示，包括：
步骤1701，获取第一训练样本，第一训练样本包括第一样本图像序列和样本接触位置。
示例性的，样本接触位置是采用三维坐标形式的坐标来表示的位置。
步骤1702，根据样本图像序列中的相邻图像内的标记图案，计算标记图案的样本差异特征。
步骤1703，调用位置估计模型对标记图案的样本差异特征进行处理，得到预测接触位置。
步骤1704，对预测接触位置和样本接触位置进行误差计算，得到第一误差损失。
步骤1705，通过莱文贝格－马夸特(Levenberg–Marquardt)反向传播算法根据第一误差损失对位置估计模型进行训练，得到训练后的位置估计模型。
在一些实施例中，以第一训练样本为图像阵列中的标记点的位移和样本接触的实际位置坐标为例进行说明，获取图像阵列中的标记点位移和样本接触位置的坐标(x 1,y 1,z 1)，将获取的图像阵列中的标记点位移输入至第一隐藏层和第一输出层，第一输出层得到样本接触位置的预测坐标(x 1’,y 1’,z 1’)，通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法对样本接触位置的坐标(x 1,y 1,z 1)和预测坐标(x 1’,y 1’,z 1’)进行处理，得到第一误差损失，根据第一误差损失对位置估计模型进行训练，得到训练后的位置估计模型。
根据本申请的实施例，前馈神经网络包括估计三维接触力的接触力估计模型，接触力估计模型包括第二隐藏层和第二输出层，该训练方法如图18所示，包括：
步骤1801，获取第二训练样本，第二训练样本包括第二样本图像序列和样本三维信息，样本三维信息是基于设置在触觉传感器尾部的力矩传感器采集到的数据进行标定得到的，三维信息包括大小和/或方向。
步骤1802，根据样本图像序列中的相邻图像内的标记图案，计算标记图案的样本差异特征。
步骤1803，调用接触力估计模型对标记图案的样本差异特征进行处理，得到预测三维信息。
步骤1804，对预测三维信息和样本三维信息进行误差计算，得到第二误 差损失。
步骤1805，通过Levenberg–Marquardt反向传播算法根据第二误差损失对接触力估计模型进行训练，得到训练后的接触力估计模型。
在一些实施例中，以第二训练样本为图像阵列中的标记点的位移和样本接触力的实际三维信息为例进行说明，获取图像阵列中的标记点的位移和样本三维信息(f x，f y，f z)，将获取的图像阵列中的标记点的位移输入至第二隐藏层和第二输出层，第二输出层则得到样本接触力的预测三维信息(f x’,f y’,f z’)，通过Levenberg–Marquardt反向传播算法对样本三维信息(f x，f y，f z)和预测三维信息(f x’,f y’,f z’)进行处理，得到第二误差损失，根据第二误差损失对位置估计模型进行训练，得到训练后的位置估计模型。
根据本申请的实施例，前馈神经网络包括用于分类接触表面的表面分类模型，表面分类模型包括第三隐藏层和第三输出层，该训练方法如图19所示，包括：
步骤1901，获取第三训练样本，第三训练样本包括第三样本图像序列和样本表面类型。
步骤1902，根据样本图像序列中的相邻图像内的标记图案，计算标记图案的样本差异特征。
步骤1903，调用表面分类模型对标记图案的样本差异特征进行处理，得到预测表面类型。
步骤1904，对预测表面类型和样本表面类型进行误差计算，得到第三误差损失。
步骤1905，通过缩放共轭梯度反向传播算法根据第三误差损失对表面分类模型进行训练，得到训练后的表面分类模型。
在一些实施例中，以第三训练样本为图像阵列中的标记点的位移和样本表面类型为例进行说明，获取图像阵列中的标记点位移和样本表面类型(S 1)，将获取的图像阵列中的标记点位移输入至第三隐藏层和第三输出层，第三输 出层得到接触表面的预测表面类型(S 1’)，通过缩放共轭梯度反向传播算法对样本表面类型(S 1)和预测表面类型(S 1’)进行处理，得到第三误差损失，根据第三误差损失对表面分类模型进行训练，得到训练后的表面分类模型。
根据本申请的实施例，上述表面分类模型的结构如图23所示，第三隐藏层可以设置一个或者两个，在一些实施例中，以第三隐藏层设置一层为例进行说明，该表面分类模型包括第三隐藏层和第三输出层，该隐藏层是基于Sigmoid隐藏神经元构建的，该第三输出层是基于Softmax隐藏神经元构建的，Sigmoid隐藏神经元适用于对物体进行分类，而Softmax隐藏神经元使得对应于接触表面的不同形状产生不同的输出的结果。
根据本申请的实施例，曲率估计模型包括：球面估计模型，球面估计模型包括第四隐藏层和第四输出层，该训练方法如图20所示，包括：
步骤2010，获取第四训练样本，第四训练样本包括第四样本图像序列和样本球面的局部曲率半径；
步骤2020，根据样本图像序列中的相邻图像内的标记图案，计算标记点的样本差异特征。
步骤2030，调用球面估计模型对标记图案的样本差异特征进行处理，得到预测曲率半径。
步骤2040，对预测曲率半径和样本球面的局部曲率半径进行误差计算，得到第四误差损失。
步骤2050，通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法根据第四误差损失对球面估计模型进行训练，得到训练后的球面估计模型。
在一些实施例中，以第四训练样本为图像阵列中的标记点的位移和样本球面的局部曲率半径为例进行说明，获取图像阵列中的标记点位移和样本球面的局部曲率半径(R 1)，将获取的图像阵列中的标记点位移输入至第四隐藏层，第四输出层得到样本球面的预测曲率半径(R 1’)，通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法对样本球面的局部曲率半径(R 1)和样 本球面的预测曲率半径(R 1’)进行处理，得到第四误差损失，根据第四误差损失对球面估计模型进行训练，得到训练后的球面估计模型。
根据本申请的实施例，曲率估计模型包括：圆柱面估计模型，圆柱面估计模型包括第五隐藏层和第五输出层，该训练方法如图21所示，包括：
步骤2101，获取第五训练样本，第五训练样本包括第五样本图像序列和样本圆柱面的局部曲率半径；
步骤2102，根据样本图像序列中的相邻图像内的标记图案，计算标记图案的差异特征。
步骤2103，调用圆柱面估计模型对标记图案的差异特征进行处理，得到预测曲率半径。
步骤2104，对预测曲率半径和样本圆柱面的局部曲率半径进行误差计算，得到第五误差损失。
步骤2105，通过Levenberg–Marquardt反向传播算法根据第五误差损失对圆柱面估计模型进行训练，得到训练后的圆柱面估计模型。
在一些实施例中，以第五训练样本为图像阵列中的标记点的位移和样本圆柱面曲率半径为例进行说明，获取图像阵列中的标记点位移和样本圆柱面的局部曲率半径(R 2)，将获取的图像阵列中的标记点位移输入至第五隐藏层，第五输出层得到样本圆柱面的预测曲率半径(R 2’)，通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法对样本圆柱面的局部曲率半径(R 2)和样本圆柱面的预测曲率半径(R 2’)进行处理，得到第五误差损失，根据第五误差损失对圆柱面估计模型进行训练，得到训练后的圆柱面估计模型。
根据本申请实施例，所接触的表面不限于球面和圆柱面，该训练样本所涉及的接触面类型和各接触面训练样本的使用个数分别如图24和图25所示。本申请选用了包括圆锥体、圆柱体、三棱锥体、三棱柱体、平面等多个形状的接触表面来训练表面分类模型，由于一个接触力大于5.5N的顶尖会在接触区域引起较大的变形，因此对于顶尖表面(如图24中的三棱锥体)，本申请 采集小于5.5N的接触力(如图25的表格所示)。
需要说明的是，对该前馈神经网络进行训练时，该前馈神经网络于未训练时所用的前馈神经网络200的内部结构一致，内部结构所实现的功能相同，因此，此处不再对训练时使用的前馈神经网络200的结构进行详细说明，具体结构见上述前馈神经网络200。
根据图2所示的触觉传感器系统的框图，该触觉传感器系统中包括上述触觉传感器以及与触觉传感器相连的芯片117。该芯片117包括可编程逻辑电路和/或程序指令，当该芯片运行时，用于执行上述基于视觉的触觉测量方法。
根据图2所示的触觉传感器系统300，该系统由上述触觉传感器和芯片117组成，该芯片117与上述基于视觉的触觉测量装置的结构中的芯片为同一类型的芯片117，该芯片117包括可编程逻辑电路和/或程序指令，当该芯片运行时，用于执行上述基于视觉的触觉测量方法。
根据本申请的实施例，上述样本接触位置的获取方式可见图26，图26示出了一个示例性实施例提供的力/力矩传感器测量接触点的几何模型。通过安装在触觉传感器下方的力矩传感器116来获得实际接触位置，在触觉传感器上建立虚拟的空间坐标系，触觉传感器测量力f和由接触力产生的力矩m落在触觉传感器建立的坐标系上，在上述传感面111在只有一个接触点的情况下，力f为接触力，而力矩m可以写成如下所示：
Figure PCTCN2020085608-appb-000001
其中r是三维向量即接触点相对于指尖的坐标系的位置。
Figure PCTCN2020085608-appb-000002
是3×3的反对称矩阵，用来表示叉乘。因为矩阵
Figure PCTCN2020085608-appb-000003
的秩为二，所以公式的解可以写成如下所示：
Figure PCTCN2020085608-appb-000004
其中
Figure PCTCN2020085608-appb-000005
是
Figure PCTCN2020085608-appb-000006
的伪逆，c是待确定的系数。由于
Figure PCTCN2020085608-appb-000007
是一个秩为二且
Figure PCTCN2020085608-appb-000008
的奇异矩阵，所以f是公式的齐次解。几何上，该公式表示在触觉传感器上建立的坐标系下的一条直线，该直线具有与传感面111的交点，如图26所示。因此，可以根据已知传感面111表面几何形状，求出交点以及交点处的法向。因为触觉传感器测量力f是传感面111上接收的压力，所以触觉传感器测量力f应该与向内的法向有一个正的内积。
在本申请实施例的一个实例中，使用整个球体代表传感面111，因此，直线与传感面111就有两个交点。其中一点的内法向与触觉传感器测量力f的内积是正的即为实际接触位置，而且位置必然也在真实的传感面111上。而另一个交点，可能落在传感面111上，也可能不落在传感面111上，但该另一个交点可以省去，因为该另一个交点与法向的内积为负。
根据本申请的实施例，对位置估计模型进行训练，训练结果如图27所示。根据图27可以看出，位置估计模型的相关系数(R)的值接近1，均方根误差(RMSE)约为0.6mm，这意味着训练后的位置估计模型与输入数据的一致性是高度相关的。
根据本申请的实施例，对接触力估计模型进行训练，训练结果如图28所示。根据图28可以看出，接触力估计模型在1.5N到8N动态力的范围内表现结果如下：经过训练，相关系数(R)的值接近1，均方根误差(RMSE)约为0.25N，这意味着训练后的接触力估计模型与输入数据的一致性是高度相关的。
根据本申请的实施例，对表面分类模型进行训练，训练结果的正确率如图29所示。利用混淆矩阵(Confusion矩阵)对表面分类模型的正确率进行评价，从图29可以看出，表面分类模型的整体的正确率在91％，这意味着训练后的表面分类模型的分类功能的正确率较高。
根据本申请的实施例，对球面估计模型进行训练，训练结果如图30所示。根据图30可以看出，球面估计模型的训练结果如下：经过训练相关系数(R)的值约为0.9，均方根误差(RMSE)约为8mm，这意味着训练后的球面估计 模型与输入数据的一致性是高度相关的。
根据本申请的实施例，对圆柱面估计模型进行训练，训练结果如图31所示。根据图31可以看出，球面估计模型的训练结果如下，经过训练相关系数(R)的值约为0.9，均方根误差(RMSE)约为10mm，这意味着训练后的圆柱面估计模型与输入数据的一致性是高度相关的。
根据本申请的实施例，上述触觉传感器系统300的使用方法的流程图如图32所示，触觉传感器上设置有传感面111，传感面的内表面上设置有标记点112，与传感面111的内表面相对的图像传感组件115，通过图像传感组件115采集柔性传感器111的内表面所形成的图像序列上标记点112的位移，将标记点112的位移作为特征值输入至上述前馈神经网络，分别测量接触点位置、三维接触力的大小和/或方向以及接触表面的局部曲率半径。另外，还可以通过力矩传感器测量的接触位置的真实值、接触力的真实值以及接触表面的局部曲率半径的真实值与触觉传感器测量的测量值比较来对上述前馈神经网络进行训练。
根据本申请的实施例，该基于视觉的触觉测量装置的结构示意图即上述上述触觉传感器与芯片相连。该基于视觉的触觉测量装置包括：第一获取模块311、第一计算模块312和前馈神经网络313，如图33所示。
第一获取模块311，用于获取图像传感组件115对传感面采集的图像序列，图像序列的图像中包括标记图案。
第一计算模块312，用于根据图像序列中的相邻图像内的标记图案112的位置，计算标记图案的差异特征。
前馈神经网络313，用于对标记图案112的差异特征进行处理，得到触觉测量结果。
在一个可选的实施例中，该前馈神经网络内设置有隐藏层和输出层，该隐藏层，用于对标记图案的差异特征进行特征提取，得到特征表示；该输出层，用于对特征表示进行处理，得到触觉测量结果。隐藏层内设置有n个隐 藏神经元，n为整数，隐藏层是基于Sigmoid隐藏神经元构建的；输出层是基于Softmax输出神经元或线性输出神经元构建的。
在一个可选的实施例中，上述触觉测量结果包括接触位置；前馈神经网络包括：位置估计模型，位置估计模型包括第一隐藏层和第一输出层，该第一隐藏层，用于述标记图案的差异特征进行特征提取，得到接触位置特征表示；该第一输出层，用于对接触位置特征表示进行处理，得到接触位置。
在一个可选的实施例中，上述触觉测量结果包括接触力的三维信息，前馈神经网络包括接触力估计模型，接触力估计模型包括：第二隐藏层和第二输出层，该第二隐藏层，用于对标记图案的差异特征进行特征提取，得到接触力特征表示；第二输出层，用于对接触力特征表示进行处理，得到接触力的三维信息，三维信息包括大小和方向中的至少一种。
在一个可选的实施例中，上述触觉测量结果包括接触表面的局部曲率半径，前馈神经网络包括：表面分类模型和至少两个曲率估计模型；该表面分类模型，用于对标记图案的差异特征进行表面识别，得到接触表面的表面类型；至少两个曲率估计模型中的目标曲率估计模型，用于根据表面类型对接触表面进行曲率预测，得到接触表面的局部曲率半径。
在一个可选的实施例中，曲率估计模型包括：球面估计模型和圆柱面估计模型，该球面估计模型，用于当表面类型是球面时，对球面进行第一曲率预测，得到球面的局部曲率半径，当表面类型是圆柱面时，圆柱面估计模型用于对圆柱面进行第二曲率预测，得到圆柱面的局部曲率半径。
在一个可选的实施例中，表面分类模型包括第三隐藏层和第三输出层，该第三隐藏层，用于对标记图案的差异特征进行表面识别，得到表面类型特征表示；第三输出层，用于对表面类型特征表示进行处理，得到接触表面的表面类型。
在一个可选的实施例中，球面估计模型包括第四隐藏层和第四输出层，该第四隐藏层，用于对球面进行第一曲率预测，得到球面曲率预测特征表示；第四输出层，用于对球面曲率预测特征表示进行处理，得到球面的局部曲率 半径。
在一个可选的实施例中，圆柱面估计模型包括第五隐藏层和第五输出层，该第五隐藏层，用于对圆柱面进行第二曲率预测，得到圆柱面曲率预测特征表示；第五输出层，用于对圆柱面曲率预测特征表示进行处理，得到圆柱面的局部曲率半径。
在一个可选的实施例中，第一计算单元，用于在图像序列中相邻的第i帧图像和第i+1帧图像中，确定最接近的两个标记图案为相同的标记图案；第二计算单元，用于根据标记图案在第i帧图像和第i+1帧图像中的位置(或，位置和形变)，计算标记图案的差异特征。
在一个可选的实施例中，该标记图案包括至少两个标记点，该标记图案的样本差异特征包括该标记点的位移该和形变中的至少一种。
在一个可选的实施例中，该标记图案包括网格，该标记图案的差异特征包括该网格中网格点的位移该和网格线的形变中的至少一种。
图34示出了一个示例性实施例提供的基于视觉的触觉测量装置的训练模块的结构示意图。
根据本申请的实施例，该基于视觉的触觉测量装置的训练模块，如图26所示，包括：
第二获取模块411，用于获取训练样本，训练样本包括样本图像序列和样本触觉结果，样本图像序列是触觉传感器内的图像传感组件采集的图像序列，触觉传感器包括传感面和图像传感组件，传感面设置有标记图案，图像序列的图像中包括标记图案。
第二计算模块412，用于根据样本图像序列中的相邻图像内的标记图案，计算标记图案的样本差异特征。
前馈神经网络模型413，用于对标记图案的样本差异特征进行处理，得到预测触觉结果，其中，前馈神经网络中的隐藏层数量少于阈值。
误差计算模块414，用于对预测触觉结果和样本触觉结果进行误差计算， 得到误差损失。
训练模块415，用于通过误差反向传播算法根据误差损失对该前馈神经网络模型进行训练，得到训练后的前馈神经网络模型。
在一个可选的实施例中，前馈神经网络模型内设置有隐藏层和输出层，该隐藏层用于对标记图案的差异特征进行特征提取，得到特征表示；输出层用于对特征表示进行处理，得到触觉测量结果。隐藏层内设置有n个隐藏神经元，n为整数，该隐藏层是基于Sigmoid隐藏神经元构建的；输出层是基于Softmax输出神经元或线性输出神经元构建的。
在一个可选的实施例中，前馈神经网络模型包括用于估计接触位置的位置估计模型，第二获取模块411，用于获取第一训练样本，第一训练样本包括第一样本图像序列和样本接触位置。估计模型用于对标记图案的样本差异特征进行处理，得到预测接触位置。误差计算模块414，还用于对所述预测接触位置和所述样本接触位置进行误差计算，得到第一误差损失。训练模块415，还用于通过Levenberg–Marquardt反向传播算法根据第一误差损失对位置估计模型进行训练，得到训练后的位置估计模型。
在一个可选的实施例中，前馈神经网络模型包括用于估计三维接触力的接触力估计模型，第二获取模块411，还用于获取第二训练样本，第二训练样本包括第二样本图像序列和样本接触力的三维信息，样本接触力的三维信息是基于设置在触觉传感器尾部的力矩传感器采集到的数据进行标定得到的，三维信息包括大小和/或方向。接触力估计模型用于对所述标记图案的样本差异特征进行处理，得到预测三维信息。误差计算模块414，还用于对预测三维信息和样本三维信息进行误差计算，得到第二误差损失。训练模块415，还用于通过Levenberg–Marquardt反向传播算法根据第二误差损失对接触力估计模型进行训练，得到训练后的接触力估计模型。
在一个可选的实施例中，前馈神经网络模型包括用于分类接触表面的表面分类模型，第二获取模快411，还用于获取第三训练样本，第三训练样本包括第三样本图像序列和样本接触表面的类型。表面分类模型用于对所述标 记图案的样本差异特征进行处理，得到预测表面类型。误差计算模块414，还用于对所述预测表面类型和所述样本表面类型进行误差计算，得到第三误差损失。训练模块415，还用于通过缩放共轭梯度反向传播算法根据第三误差损失对表面分类模型进行训练，得到训练后的表面分类模型。
在一个可选的实施例中，前馈神经网络模型包括至少两个用于估计曲率的曲率估计模型，第二获取模块411，用于获取训练样本，训练样本包括样本表面的类型和样本触觉结果；第二计算模块412，用于通过误差反向传播算法根据误差损失对前馈神经网络进行训练，得到训练后的前馈神经网络。
在一个可选的实施例中，曲率估计模型包括：球面估计模型，第二获取模快411，用于获取第四训练样本，第四训练样本包括第四样本表面的类型和样本球面的局部曲率半径。球面估计模型用于当所述预测表面类型是球面时，对所述标记图案的样本差异特征进行处理，得到预测曲率半径。误差计算模块414，还用于对球面的预测曲率半径和样本球面的局部曲率半径进行误差计算，得到第四误差损失。训练模块415，还用于通过Levenberg–Marquardt反向传播算法根据第四误差损失对球面估计模型进行训练，得到训练后的球面估计模型。
在一个可选的实施例中，曲率估计模型包括：圆柱面估计模型，第二获取模快411，用于获取第五训练样本，第五训练样本包括第五样本表面的类型和样本圆柱面的局部曲率半径。圆柱面估计模型，用于当预测表面类型是圆柱面时，调用圆柱面估计模型对标记图案的差异特征进行处理，得到预测曲率半径。误差计算模块414，还用于对圆柱的预测曲率半径和样本圆柱面的局部曲率半径进行误差计算，得到第五误差损失。训练模块415，还用于通过Levenberg–Marquardt反向传播算法根据第五误差损失对圆柱面估计模型进行训练，得到训练后的圆柱面估计模型。
本申请的实施例还提供了一种计算机设备，该计算机设备包括存储器和处理器，存储器存储有计算机可读指令，处理器执行计算机可读指令时实现 上述前馈神经网络的训练方法的步骤。
图35示出了本申请一个示例性实施例提供的计算机设备的结构示意图。该计算机设备可以用于执行上述前馈神经网络的训练方法。具体来讲：
计算机设备3500包括中央处理单元(CPU)3501、包括随机存取存储器(RAM)3502和只读存储器(ROM)3503的系统存储器3504，以及连接系统存储器3504和中央处理单元3501的系统总线3505。计算机设备3500还包括帮助计算机内的各个器件之间传输信息的基本输入/输出系统(I/O系统)3506，和用于存储操作系统3513、应用程序3514和其他程序模块3515的大容量存储设备3507。
基本输入/输出系统3506包括有用于显示信息的显示器3508和用于用户输入信息的诸如鼠标、键盘之类的输入设备3509。其中显示器3508和输入设备3509都通过连接到系统总线3505的输入输出控制器3510连接到中央处理单元3501。基本输入/输出系统3506还可以包括输入输出控制器3510以用于接收和处理来自键盘、鼠标、或电子触控笔等多个其他设备的输入。类似地，输入输出控制器3510还提供输出到显示屏、打印机或其他类型的输出设备。
大容量存储设备3507通过连接到系统总线3505的大容量存储控制器(未示出)连接到中央处理单元3501。大容量存储设备3507及其相关联的计算机可读介质为计算机设备3500提供非易失性存储。也就是说，大容量存储设备3507可以包括诸如硬盘或者CD-ROM驱动器之类的计算机可读介质(未示出)。
不失一般性，计算机可读介质可以包括计算机存储介质和通信介质。计算机存储介质包括以用于存储诸如计算机可读指令、数据结构、程序模块或其他数据等信息的任何方法或技术实现的易失性和非易失性、可移动和不可移动介质。计算机存储介质包括RAM、ROM、EPROM、EEPROM、闪存或其他固态存储其技术，CD-ROM、DVD或其他光学存储、磁带盒、磁带、磁盘存储或其他磁性存储设备。当然，本领域技术人员可知计算机存储介质不 局限于上述几种。上述的系统存储器3504和大容量存储设备3507可以统称为存储器。
根据本申请的各种实施例，计算机设备3500还可以通过诸如因特网等网络连接到网络上的远程计算机运行。也即计算机设备3500可以通过连接在系统总线3505上的网络接口单元3511连接到网络3512，或者说，也可以使用网络接口单元3511来连接到其他类型的网络或远程计算机系统(未示出)。
上述存储器还包括一个或者一个以上的程序，一个或者一个以上程序存储于存储器中，被配置由CPU执行。
本申请的实施例还提供了一种计算机可读存储介质，存储有计算机可读指令，其特征在于，计算机可读指令被处理器执行时实现上述基于视觉的触觉测量方法，或者，上述前馈神经网络的训练方法。
本申请的实施例还提供了一种机器人系统，该机器人系统包括：芯片和触觉传感器，触觉传感器设置在指尖部位和皮肤部位中的至少一个部位，触觉传感器包括柔性传感面和朝向柔性传感面的内表面设置的图像传感组件，柔性传感面内设置有标记点，图像传感组件与芯片相连，该芯片包括可编程逻辑电路和程序指令中的至少一种，当芯片运行时，用于执行上述基于视觉的触觉测量方法。
本申请的实施例还提供了一种触觉传感器系统300的使用方法流程图，如图32所示，该使用方法适用于上述触觉传感器、触觉传感器中的前馈神经网络、上述前馈神经网络中的位置估计模型、接触力估计模型、表面分类模型、球面估计曲率模型、圆柱面估计曲率模型、上述触觉传感器系统和上述机器人系统。
以上所述仅为本申请的较佳实施例，并不用以限制本申请，凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。

Claims (31)

  1. 一种基于视觉的触觉测量方法，所述方法由芯片执行，所述芯片与触觉传感器相连，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案；
    所述方法包括：
    获取所述图像传感组件对所述传感面采集的图像序列，所述图像序列的图像中包括所述标记图案；
    根据所述图像序列中的相邻图像内的所述标记图案，计算所述标记图案的差异特征；及
    调用前馈神经网络对所述标记图案的差异特征进行处理，得到触觉测量结果；其中，所述前馈神经网络内的隐藏层数量小于阈值。
  2. 根据权利要求1所述的方法，其特征在于，所述前馈神经网络内设置有所述隐藏层和输出层；所述调用前馈神经网络对所述标记图案的差异特征进行处理，得到触觉测量结果，包括：
    调用所述前馈神经网络中的所述隐藏层对所述标记图案的差异特征进行特征提取，得到特征表示；及
    调用所述前馈神经网络中的所述输出层对所述特征表示进行处理，得到触觉测量结果。
  3. 根据权利要求2所述的方法，其特征在于，所述隐藏层内设置有n个隐藏神经元，n为整数；
    所述隐藏层是基于逻辑斯蒂函数Sigmoid隐藏神经元构建的；及
    所述输出层是基于归一化指数函数Softmax输出神经元或线性输出神经元构建的。
  4. 根据权利要求2所述的方法，其特征在于，所述触觉测量结果包括接触位置；所述前馈神经网络包括：位置估计模型，所述位置估计模型包括第一隐藏层和第一输出层；
    所述调用所述前馈神经网络中的所述隐藏层对所述标记图案的差异特征 进行特征提取，得到特征表示，包括：
    调用所述位置估计模型中的所述第一隐藏层对所述标记图案的差异特征进行特征提取，得到接触位置特征表示；
    所述调用所述前馈神经网络中的所述输出层对所述特征表示进行处理，得到触觉测量结果，包括：
    调用所述位置估计模型中的所述第一输出层对所述接触位置特征表示进行处理，得到所述接触位置。
  5. 根据权利要求2所述的方法，其特征在于，所述触觉测量结果包括接触力的三维信息，所述前馈神经网络包括接触力估计模型，所述接触力估计模型包括：第二隐藏层和第二输出层；
    所述调用所述前馈神经网络中的所述隐藏层对所述标记图案的差异特征进行特征提取，得到特征表示，包括：
    调用所述接触力估计模型中的所述第二隐藏层对所述标记图案的差异特征进行特征提取，得到接触力特征表示；
    所述调用所述前馈神经网络中的所述输出层对所述特征表示进行处理，得到触觉测量结果，包括：
    调用所述接触力估计模型中的所述第二输出层对所述接触力特征表示进行处理，得到所述接触力的三维信息，所述三维信息包括大小和方向中的至少一种。
  6. 根据权利要求2所述的方法，其特征在于，所述触觉测量结果包括接触表面的局部曲率半径；所述前馈神经网络包括：表面分类模型和至少两个曲率估计模型；
    所述调用前馈神经网络对所述标记图案的差异特征进行处理，得到触觉测量结果，包括：
    调用所述表面分类模型对所述标记图案的差异特征进行表面识别，得到接触表面的表面类型；及
    根据所述表面类型调用所述至少两个曲率估计模型中的目标曲率估计模 型对所述接触表面进行曲率预测，得到所述接触表面的局部曲率半径。
  7. 根据权利要求6所述的方法，其特征在于，所述曲率估计模型包括：球面估计模型和圆柱面估计模型；所述根据所述表面类型调用所述至少两个曲率估计模型中的目标曲率估计模型对所述接触表面进行曲率预测，得到所述接触表面的局部曲率半径，包括：
    当所述表面类型是球面时，调用所述球面估计模型对所述球面进行第一曲率预测，得到所述球面的局部曲率半径；及
    当所述表面类型是圆柱面时，调用所述圆柱面估计模型对所述圆柱面进行第二曲率预测，得到所述圆柱面的局部曲率半径。
  8. 根据权利要求6所述的方法，其特征在于，所述表面分类模型包括第三隐藏层和第三输出层；所述调用所述表面分类模型对所述标记图案的差异特征进行表面识别，得到接触表面的表面类型，包括：
    调用所述表面分类模型中的所述第三隐藏层对所述标记图案的差异特征进行表面识别，得到表面类型特征表示；及
    调用所述表面分类模型中的所述第三输出层对所述表面类型特征表示进行处理，得到接触表面的表面类型。
  9. 根据权利要求7所述的方法，其特征在于，所述球面估计模型包括第四隐藏层和第四输出层；所述调用所述球面估计模型对所述球面进行第一曲率预测，得到所述球面的局部曲率半径，包括：
    调用所述球面估计模型中的第四隐藏层对所述球面进行第一曲率预测，得到球面曲率预测特征表示；及
    调用所述球面估计模型中的第四输出层对所述球面曲率预测特征表示进行处理，得到所述球面的局部曲率半径。
  10. 根据权利要求7所述的方法，其特征在于，所述圆柱面估计模型包括第五隐藏层和第五输出层；所述调用所述圆柱面估计模型对所述圆柱面进行第二曲率预测，得到所述圆柱面的局部曲率半径，包括：
    调用所述圆柱面估计模型中的第五隐藏层对所述圆柱面进行第二曲率预 测，得到圆柱面曲率预测特征表示；及
    调用所述圆柱面估计模型中的第五输出层对所述圆柱面曲率预测特征表示进行处理，得到所述圆柱面的局部曲率半径。
  11. 根据权利要求1至10任一所述的方法，其特征在于，所述标记图案包括至少两个标记点，所述标记图案的差异特征包括所述标记点的位移和形变中的至少一种。
  12. 根据权利要求1至10任一项所述的方法，其特征在于，所述标记图案包括网格，所述标记图案的差异特征包括所述网格中网格点的位移和网格线的形变中的至少一种。
  13. 根据权利要求1至10任一所述的方法，其特征在于，所述根据所述图像序列中的相邻图像内的所述标记图案，计算所述标记图案的差异特征，包括：
    在所述图像序列中相邻的第i帧图像和第i+1帧图像中，确定最接近的两个标记图案为相同的标记图案，i为整数；及
    根据所述标记图案在所述第i帧图像和所述第i+1帧图像中的位置和形变中的至少一种，计算所述标记图案的差异特征。
  14. 一种前馈神经网络的训练方法，由计算机设备执行，所述方法包括：
    获取训练样本，所述训练样本包括样本图像序列和样本触觉结果，所述样本图像序列是触觉传感器内的图像传感组件采集的图像序列，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案，所述图像序列的图像中包括所述标记图案；
    根据所述样本图像序列中的相邻图像内的所述标记图案，计算所述标记图案的样本差异特征；
    调用前馈神经网络对所述标记图案的样本差异特征进行处理，得到预测触觉结果；所述前馈神经网络中的隐藏层数量少于阈值；
    对所述预测触觉结果和所述样本触觉结果进行误差计算，得到误差损失； 及
    通过误差反向传播算法根据所述误差损失对所述前馈神经网络进行训练，得到训练后的前馈神经网络。
  15. 根据权利要求14所述的方法，其特征在于，所述前馈神经网络内设置有所述隐藏层和输出层；所述调用前馈神经网络对所述标记图案的样本差异特征进行处理，得到预测触觉结果，包括：
    调用所述前馈神经网络中的所述隐藏层对所述标记图案的样本差异特征进行特征提取，得到特征表示；及
    调用所述前馈神经网络中的所述输出层对所述特征表示进行处理，得到预测触觉结果。
  16. 根据权利要求15所述的方法，其特征在于，所述隐藏层内设置有n个隐藏神经元，n为整数；
    所述隐藏层是基于逻辑斯蒂函数Sigmoid隐藏神经元构建的；及
    所述输出层是基于归一化指数函数Softmax输出神经元或线性输出神经元构建的。
  17. 根据权利要求14所述的方法，其特征在于，所述前馈神经网络包括用于估计接触位置的位置估计模型，所述获取训练样本，所述训练样本包括样本图像序列和样本触觉结果，包括：
    获取第一训练样本，所述第一训练样本包括第一样本图像序列和样本接触位置；
    所述调用前馈神经网络对所述标记图案的样本差异特征进行处理，得到预测触觉结果，包括：
    调用所述位置估计模型对所述标记图案的样本差异特征进行处理，得到预测接触位置；
    所述对所述预测触觉结果和所述样本接触位置进行误差计算，得到误差损失，包括：
    对所述预测接触位置和所述样本接触位置进行误差计算，得到第一误差 损失；
    所述通过误差反向传播算法根据所述误差损失对所述前馈神经网络进行训练，得到训练后的前馈神经网络，包括：
    通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法根据所述第一误差损失对所述位置估计模型进行训练，得到训练后的位置估计模型。
  18. 根据权利要求14所述的方法，其特征在于，所述前馈神经网络包括用于估计接触力的三维信息的接触力估计模型，所述获取训练样本，所述训练样本包括样本图像序列和样本触觉结果，包括：
    获取第二训练样本，所述第二训练样本包括第二样本图像序列和样本三维信息，所述样本三维信息是基于设置在所述触觉传感器尾部的力矩传感器采集到的数据进行标定得到的，所述样本三维信息包括大小和方向中的至少一种；
    所述调用前馈神经网络对所述标记图案的样本差异特征进行处理，得到预测触觉结果，包括：
    调用所述接触力估计模型对所述标记图案的样本差异特征进行处理，得到预测三维信息；
    所述对所述预测触觉结果和所述样本触觉结果进行误差计算，得到误差损失，包括：
    对所述预测三维信息和所述样本三维信息进行误差计算，得到第二误差损失；
    所述通过误差反向传播算法根据所述误差损失对所述前馈神经网络进行训练，得到训练后的前馈神经网络，包括：
    通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法根据所述第二误差损失对所述接触力估计模型进行训练，得到训练后的接触力估计模型。
  19. 根据权利要求14所述的方法，其特征在于，所述前馈神经网络包括用于分类接触表面的表面分类模型；所述获取训练样本，所述训练样本包括样本图像序列和样本触觉结果，包括：
    获取第三训练样本，所述第三训练样本包括第三样本图像序列和样本表面类型；
    所述调用前馈神经网络对所述标记图案的样本差异特征进行处理，得到预测触觉结果，包括：
    调用所述表面分类模型对所述标记图案的样本差异特征进行处理，得到预测表面类型；
    所述对所述预测触觉结果和所述样本触觉结果进行误差计算，得到误差损失，包括：
    对所述预测表面类型和所述样本表面类型进行误差计算，得到第三误差损失；
    所述通过误差反向传播算法根据所述误差损失对所述前馈神经网络进行训练，得到训练后的前馈神经网络，包括：
    通过缩放共轭梯度反向传播算法根据所述第三误差损失对所述表面分类模型进行训练，得到训练后的表面分类模型。
  20. 根据权利要求19所述的方法，其特征在于，所述前馈神经网络还包括曲率估计模型，所述曲率估计模型包括球面估计模型，所述方法还包括：
    获取第四训练样本，所述第四训练样本包括第四样本图像序列和样本球面的局部曲率半径；
    当所述预测表面类型是球面时，调用所述调用球面估计模型对所述标记图案的样本差异特征进行处理，得到预测曲率半径；
    对所述球面的预测曲率半径和所述样本球面的局部曲率半径进行误差计算，得到第四误差损失；及
    通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法根据所述第四误差损失对所述球面估计模型进行训练，得到训练后的球面估计模型。
  21. 根据权利要求19所述的方法，其特征在于，所述前馈神经网络还包括曲率估计模型，所述曲率估计模型包括圆柱面估计模型，所述方法还包括：
    获取第五训练样本，所述第五训练样本包括第五样本图像序列和样本圆 柱面的局部曲率半径；
    当所述预测表面类型是圆柱面时，调用所述圆柱面估计模型对所述标记图案的差异特征进行处理，得到预测曲率半径；
    对所述圆柱的预测曲率半径和所述样本圆柱面的局部曲率半径进行误差计算，得到第五误差损失；及
    通过莱文贝格－马夸特Levenberg–Marquardt反向传播算法根据所述第五误差损失对所述圆柱面估计模型进行训练，得到训练后的圆柱面估计模型。
  22. 根据权利要求14至21任一所述的方法，其特征在于，所述标记图案包括至少两个标记点，所述标记图案的样本差异特征包括所述标记点的位移和形变中的至少一种。
  23. 根据权利要求14至21任一项所述的方法，其特征在于，所述标记图案包括网格，所述标记图案的样本差异特征包括：所述网格中网格点的位移和网格线的形变中的至少一种。
  24. 一种基于视觉的触觉测量装置，其特征在于，所述装置应用于芯片中，所述芯片与触觉传感器相连，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案；
    所述装置包括：
    第一获取模块，用于获取所述图像传感组件对所述传感面采集的图像序列，所述图像序列的图像中包括所述标记图案；
    第一计算模块，用于根据所述图像序列中的相邻图像内的所述标记图案，计算所述标记图案的差异特征；及
    前馈神经网络，用于对所述标记图案的差异特征进行处理，得到触觉测量结果；其中，所述前馈神经网络内的隐藏层数量小于阈值。
  25. 一种前馈神经网络的训练装置，其特征在于，所述装置包括：
    第二获取模块，用于获取训练样本，所述训练样本包括样本图像序列和 样本触觉结果，所述样本图像序列是触觉传感器内的图像传感组件采集的图像序列，所述触觉传感器包括传感面和图像传感组件，所述传感面设置有标记图案，所述图像序列的图像中包括所述标记图案；
    第二计算模块，用于根据所述样本图像序列中的相邻图像内的所述标记图案，计算所述标记图案的样本差异特征；
    前馈神经网络模型，用于对所述标记图案的样本差异特征进行处理，得到预测触觉结果；所述前馈神经网络中的隐藏层数量少于阈值；
    误差计算模块，用于对所述预测触觉结果和所述样本触觉结果进行误差计算，得到误差损失；及
    训练模块，用于通过误差反向传播算法根据所述误差损失对所述前馈神经网络进行训练，得到训练后的前馈神经网络。
  26. 一种芯片，其特征在于，所述芯片包括可编程逻辑电路和程序指令中的至少一种，当所述芯片运行时所述芯片用于执行如权利要求1至13任一所述的基于视觉的触觉测量方法。
  27. 一种触觉传感器系统，其特征在于，所述系统包括：触觉传感器和芯片，所述触觉传感器包括柔性传感面和图像传感组件，所述传感面设置有标记图案，所述图像传感组件与所述芯片相连；
    所述芯片包括可编程逻辑电路和程序指令中的至少一种，当所述芯片运行时所述芯片用于执行如权利要求1至13任一所述的基于视觉的触觉测量方法。
  28. 一种计算机设备，包括存储器和处理器，所述存储器存储有计算机可读指令，其特征在于，所述处理器执行所述计算机可读指令时实现权利要求14至23中任一项所述的前馈神经网络的训练方法的步骤。
  29. 一种计算机可读存储介质，存储有计算机可读指令，其特征在于，所述计算机可读指令被处理器执行时实现权利要求1至13中任一项所述的基于视觉的触觉测量方法的步骤。
  30. 一种计算机可读存储介质，存储有计算机可读指令，其特征在于，所述计算机可读指令被处理器执行时实现权利要求14至23中任一项所述的前馈神经网络的训练方法的步骤。
  31. 一种机器人系统，其特征在于，所述机器人系统包括：芯片和触觉传感器，所述触觉传感器设置在指尖部位和皮肤部位中的至少一个部位，所述触觉传感器包括柔性传感面和朝向所述柔性传感面的内表面设置的图像传感组件，所述柔性传感面内设置有标记图案，所述图像传感组件与所述芯片相连；
    所述芯片包括可编程逻辑电路和程序指令中的至少一种，当所述芯片运行时所述芯片用于执行权利要求1至14任一所述的基于视觉的触觉测量方法。
PCT/CN2020/085608 2019-05-16 2020-04-20 基于视觉的触觉测量方法、装置、芯片及存储介质 Ceased WO2020228489A1 (zh)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP20804776.1A EP3971684B1 (en) 2019-05-16 2020-04-20 Vision-based tactile measurement method, apparatus, chip and storage medium
US17/369,837 US12214487B2 (en) 2019-05-16 2021-07-07 Vision-based tactile measurement method and apparatus, chip, and storage medium
US19/003,999 US20260115934A1 (en) 2019-05-16 2024-12-27 Vision-based tactile measurement method and apparatus, chip, and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910411693.6 2019-05-16
CN201910411693.6A CN110162175B (zh) 2019-05-16 2019-05-16 基于视觉的触觉测量方法、装置、设备及存储介质

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/369,837 Continuation US12214487B2 (en) 2019-05-16 2021-07-07 Vision-based tactile measurement method and apparatus, chip, and storage medium

Publications (1)

Publication Number Publication Date
WO2020228489A1 true WO2020228489A1 (zh) 2020-11-19

Family

ID=67631149

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/085608 Ceased WO2020228489A1 (zh) 2019-05-16 2020-04-20 基于视觉的触觉测量方法、装置、芯片及存储介质

Country Status (4)

Country Link
US (2) US12214487B2 (zh)
EP (1) EP3971684B1 (zh)
CN (2) CN110162175B (zh)
WO (1) WO2020228489A1 (zh)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112949197A (zh) * 2021-03-08 2021-06-11 北京理工大学 一种基于深度学习的平凸透镜曲率半径在线测量方法
WO2022148542A1 (en) * 2021-01-08 2022-07-14 MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. Method for force inference, method for training a feed-forward neural network, force inference module, and sensor arrangement
WO2022191910A1 (en) * 2021-03-08 2022-09-15 Massachusetts Institute Of Technology Enhanced depth estimation using deep learning
CN121068067A (zh) * 2025-11-06 2025-12-05 哈尔滨工业大学 视触觉传感器
US12620107B2 (en) 2022-01-06 2026-05-05 Massachusetts Institute Of Technology Enhanced depth estimation using deep learning

Families Citing this family (24)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110162175B (zh) * 2019-05-16 2022-04-19 腾讯科技（深圳）有限公司 基于视觉的触觉测量方法、装置、设备及存储介质
CN110722591B (zh) * 2019-10-12 2021-06-22 哈尔滨工业大学（深圳） 柔性体的力感知方法、夹爪及机械手
CN110737339B (zh) * 2019-10-28 2021-11-02 福州大学 基于深度学习的视觉—触觉交互模型构建方法
CN112060085B (zh) * 2020-08-24 2021-10-08 清华大学 一种基于视触多尺度定位的机器人操作位姿控制方法
CN112652060A (zh) * 2021-01-06 2021-04-13 上海交通大学 一种基于粒子图像测速法的多模态视触觉传感系统及方法
WO2022148541A1 (en) * 2021-01-08 2022-07-14 MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. Sensor arrangement for sensing forces and methods for fabricating a sensor arrangement and parts thereof
CN113670928B (zh) * 2021-06-29 2023-03-28 华中农业大学 一种基于视触觉的无损瓜果损伤识别与硬度检测装置
CN114179082B (zh) * 2021-12-07 2024-09-13 南京工程学院 一种基于接触力信息的图像力触觉检测装置及再现方法
CN114544052B (zh) * 2022-01-19 2023-03-28 清华大学 触觉传感器、机器人、实现触觉信息获取的方法及装置
CN114407084B (zh) * 2022-01-21 2025-03-28 广州市香港科大霍英东研究院 基于机器视觉的机器人关节形变传感器、机械臂及方法
CN116803625B (zh) * 2022-03-17 2024-11-19 腾讯科技（深圳）有限公司 机器人控制方法、装置、设备和存储介质
CN114732396B (zh) * 2022-04-02 2026-02-10 冯军峰 硬膜外和硬膜下血肿监测系统、计算机可存储介质及装置
CN114659678B (zh) * 2022-04-12 2024-11-12 深圳市松果体机器人科技有限公司 面形柔性触觉传感器
CN114769021B (zh) * 2022-04-24 2022-11-25 广东天太机器人有限公司 一种基于全角度模板识别的机器人喷涂系统及方法
CN115018700B (zh) * 2022-04-26 2025-07-01 大连理工大学 一种触觉图像超分辨率重建方法及采集系统
CN115824470B (zh) * 2022-10-25 2025-05-02 中国科学院自动化研究所 一种触觉传感器、制备方法和点云重建方法
CN115719385B (zh) * 2022-11-14 2025-10-31 北京通用人工智能研究院 基于视觉的触觉传感器标定方法及装置
CN116399500B (zh) * 2023-03-29 2026-04-21 杜承金 一种接触力检测方法及磁触觉传感器
CN116337013B (zh) * 2023-03-29 2026-01-30 北京信息科技大学 一种基于神经网络的软体操作臂的位姿测量方法
US12481403B2 (en) * 2023-04-21 2025-11-25 Samsung Electronics Co., Ltd. Continuous tactile feedback for electronic devices
CN116596022B (zh) * 2023-05-18 2025-12-09 浙江大学 一种秀丽隐杆线虫仿生多感知-运动行为方法及系统
CN118809592B (zh) * 2024-07-09 2025-09-23 深圳市优必选科技股份有限公司 机器人控制方法、装置、电子设备及存储介质
CN119473019A (zh) * 2025-01-14 2025-02-18 华东交通大学 基于改进图卷积神经网络的穿戴设备触觉感知方法及系统
CN119880231B (zh) * 2025-03-28 2025-06-06 湖南大学 一种集力觉与视触觉的多功能传感器及其测量方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110029470A1 (en) * 2009-07-30 2011-02-03 Northwestern University Systems, methods, and apparatus for reconstruction of 3-d object morphology, position, orientation and texture using an array of tactile sensors
CN106204620A (zh) * 2016-07-21 2016-12-07 清华大学 一种基于微视觉的触觉三维力检测方法
CN109716361A (zh) * 2016-09-08 2019-05-03 谷歌有限责任公司 执行触摸运动预测的深度机器学习
CN110162175A (zh) * 2019-05-16 2019-08-23 腾讯科技（深圳）有限公司 基于视觉的触觉测量方法、装置、设备及存储介质

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1057246C (zh) * 1998-07-01 2000-10-11 中国科学院合肥智能机械研究所 一种多功能类皮肤型传感器
CN1280069C (zh) * 2003-11-01 2006-10-18 中国科学院合肥智能机械研究所 一种柔性触觉传感器及触觉信息检测方法
EP3387516B1 (en) * 2015-12-09 2022-04-20 FlatFrog Laboratories AB Improved stylus identification
RU2665273C2 (ru) * 2016-06-03 2018-08-28 Автономная некоммерческая образовательная организация высшего образования "Сколковский институт науки и технологий" Обучаемые визуальные маркеры и способ их продуцирования
EP3267288A1 (en) 2016-07-08 2018-01-10 Thomson Licensing Method, apparatus and system for rendering haptic effects
CN106092382B (zh) * 2016-07-20 2018-09-11 山东大学 一种基于弹性体三维形变的触觉传感器及检测方法
CN106650918B (zh) * 2016-11-25 2019-08-30 东软集团股份有限公司 构建系统模型的方法和装置
CN108229548A (zh) * 2017-12-27 2018-06-29 华为技术有限公司 一种物体检测方法及装置
CN108692644B (zh) * 2018-03-26 2019-09-27 华中科技大学 一种复杂曲面三坐标测量装置及误差补偿方法
WO2020222027A1 (en) * 2019-04-29 2020-11-05 Ecole Polytechnique Federale De Lausanne (Epfl) Dynamic media content categorisation method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110029470A1 (en) * 2009-07-30 2011-02-03 Northwestern University Systems, methods, and apparatus for reconstruction of 3-d object morphology, position, orientation and texture using an array of tactile sensors
CN106204620A (zh) * 2016-07-21 2016-12-07 清华大学 一种基于微视觉的触觉三维力检测方法
CN109716361A (zh) * 2016-09-08 2019-05-03 谷歌有限责任公司 执行触摸运动预测的深度机器学习
CN110162175A (zh) * 2019-05-16 2019-08-23 腾讯科技（深圳）有限公司 基于视觉的触觉测量方法、装置、设备及存储介质

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
FANG, BIN ET AL.: "A Dual-Modal Vision-Based Tactile Sensor for Robotic Hand Grasping", 2018 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION (ICRA), 21 May 2018 (2018-05-21), pages 1 - 9, XP033403444, ISSN: 2577-087X *
See also references of EP3971684A4
WANG,FE ILU ET AL.: "Application of RBF neural network in decoupling research of flexible tactile sensor", COMPUTER ENGINEERING AND APPLICATIONS, vol. 51, no. 10, 31 May 2015 (2015-05-31), pages 20 - 25, XP055753127, ISSN: 1002-8331 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022148542A1 (en) * 2021-01-08 2022-07-14 MAX-PLANCK-Gesellschaft zur Förderung der Wissenschaften e.V. Method for force inference, method for training a feed-forward neural network, force inference module, and sensor arrangement
CN112949197A (zh) * 2021-03-08 2021-06-11 北京理工大学 一种基于深度学习的平凸透镜曲率半径在线测量方法
WO2022191910A1 (en) * 2021-03-08 2022-09-15 Massachusetts Institute Of Technology Enhanced depth estimation using deep learning
US12620107B2 (en) 2022-01-06 2026-05-05 Massachusetts Institute Of Technology Enhanced depth estimation using deep learning
CN121068067A (zh) * 2025-11-06 2025-12-05 哈尔滨工业大学 视触觉传感器
CN121068067B (zh) * 2025-11-06 2026-02-10 哈尔滨工业大学 视触觉传感器

Also Published As

Publication number Publication date
EP3971684B1 (en) 2026-02-18
CN110162175B (zh) 2022-04-19
CN110162175A (zh) 2019-08-23
US20260115934A1 (en) 2026-04-30
CN114661169A (zh) 2022-06-24
EP3971684A1 (en) 2022-03-23
US12214487B2 (en) 2025-02-04
EP3971684A4 (en) 2022-06-29
CN114661169B (zh) 2024-08-13
US20210334584A1 (en) 2021-10-28

Similar Documents

Publication Publication Date Title
CN110162175B (zh) 基于视觉的触觉测量方法、装置、设备及存储介质
US11565407B2 (en) Learning device, learning method, learning model, detection device and grasping system
CN108764048B (zh) 人脸关键点检测方法及装置
EP3853764A1 (en) Training neural networks for vehicle re-identification
Hoang et al. Context-aware grasp generation in cluttered scenes
JP7051751B2 (ja) 学習装置、学習方法、学習モデル、検出装置及び把持システム
JP7349423B2 (ja) 学習装置、学習方法、学習モデル、検出装置及び把持システム
CN110335314A (zh) 基于目标物体的二维图像预测目标物体实时位姿的方法
US12524087B2 (en) Pen state detection circuit, system, and method
CN115082511A (zh) 一种基于轻量级堆叠沙漏网络的机械臂姿态估计方法
CN114399628A (zh) 复杂空间环境下的绝缘子高效检测系统
Wang et al. Hierarchical viewpoint planning for complex surfaces in industrial product inspection
Hu et al. Active shape reconstruction using a novel visuotactile palm sensor
Salah et al. NeuTac: Zero-shot Sim2Real measurement for neuromorphic vision-based tactile sensors
Yu et al. Mtgrasp: Multiscale 6-DoF robotic grasp detection
Liang et al. AllTact fin ray: A compliant robot gripper with omni-directional tactile sensing
CN115331021B (zh) 基于多层特征自身差异融合的动态特征提取与描述方法
JP7254262B2 (ja) 作業推定装置、作業推定方法、及び、作業推定プログラム
Daqi et al. An industrial intelligent grasping system based on convolutional neural network
Chu et al. ViTac-Gripper: A Vision-Based Tactile Gripper With Enhanced Multi-Physical Field Perception for Underwater Robots
Tu et al. Contact localization from soft tactile array sensor using tactile image
Luo et al. Efficacy of varying sensing features for enhanced performance of deep-learning-informed multidimensional force platform
CN121936103A (zh) 视触觉传感器的仿真方法及装置、存储介质、程序产品、计算设备
CN120706215A (zh) 抓取参数模型训练方法、装置、电子设备和存储介质
KR20240059519A (ko) 치수 측정 장치 및 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20804776

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

WWE Wipo information: entry into national phase

Ref document number: 2020804776

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2020804776

Country of ref document: EP

Effective date: 20211216

WWG Wipo information: grant in national office

Ref document number: 2020804776

Country of ref document: EP