CN115290077A - Positioning method, positioning device, electronic equipment and storage medium - Google Patents

Abstract

The invention belongs to the technical field of positioning, and discloses a positioning method, a device, an electronic device and a storage medium. The positioning method includes: a camera arranged on a traveling vehicle obtains continuously photographed images; parameters, calculate the first mileage of the vehicle; according to the first mileage calculation failure data output by the encoder set on the vehicle, calculate the second mileage of the vehicle; according to the first mileage and the second mileage, determine the the current position of the vehicle. The invention can realize high-precision positioning in the tunnel, and provides a method of using encoder positioning when the first mileage calculation fails, thereby improving the success rate of positioning. The invention provides a vehicle positioning method suitable for a tunnel, and solves the problems of inaccurate positioning or positioning failure in the traditional vehicle positioning method in the tunnel.

Description

A positioning method, device, electronic equipment and storage medium

technical field

The invention belongs to the technical field of positioning, and in particular relates to a positioning method, device, electronic equipment and storage medium.

Background technique

As an important basic part of transportation lines and urban underground space, tunnels play a positive role in facilitating people's travel and relieving traffic pressure.

Due to the complexity and difference of meteorological, hydrological and geographical environments, as well as the limitations of the survey and design technical means and construction technology level during tunnel construction, coupled with the loss during operation, many tunnels have various degrees of disease, affecting Tunnel healthy operation. Tunnel disease refers to the damage or failure of the tunnel lining, road surface, portal and auxiliary structures caused by physical or chemical reasons, resulting in damage to the tunnel's aesthetics, structural bearings, and use functions.

In view of the low efficiency of manual detection, many current tunnel defect detection methods use multi-camera arrays to collaboratively measure large-section images of the lining. This method generally uses a vehicle-mounted camera array to travel through the tunnel to obtain large-section images of the lining, and then identify and detect tunnel defects based on the captured images. At the same time, in order to further realize the manual analysis and maintenance of the disease, it is also necessary to determine the position of the disease in the tunnel. Specifically, the specific location of the disease in the tunnel can be determined according to the current location of the vehicle in the tunnel when the image is taken.

However, in actual operation, the tunnel is a semi-enclosed environment, and often lacks satellite positioning signals or network signals. Therefore, the traditional GPS (Global Positioning System, Global Positioning System) method or network positioning method is used to locate the position of the vehicle. Often, the positioning is inaccurate or fails, which makes it difficult to further determine the location of the disease in the tunnel.

Therefore, there is a need for a vehicle positioning method suitable for tunnels, so as to solve the problem of inaccurate positioning or positioning failure of traditional vehicle positioning methods in tunnels.

Through the above analysis, the problems and defects in the prior art are:

The existing technology has low accuracy of tunnel disease information or image recognition and positioning due to damage or failure of tunnel lining, road surface, portal and auxiliary structures caused by physical or chemical reasons, resulting in tunnel beauty, structural load-bearing, and use function damage.

Contents of the invention

Embodiments of the present invention provide a positioning method, device, electronic equipment and storage medium to solve the problem of inaccurate positioning or positioning failure in traditional vehicle positioning methods in tunnels.

In view of this, the first aspect of the embodiment of the present invention provides a positioning method, the method includes: the camera installed on the driving vehicle acquires images taken continuously, and calculates the vehicle driving position according to the currently acquired images and the calibration parameters of the camera. the first mileage of

Calculate the second mileage traveled by the vehicle according to the first mileage calculation failure data output by the encoder installed on the vehicle;

The current location of the vehicle is determined according to the first mileage and the second mileage.

Optionally, the positioning method also includes:

Acquiring a plurality of first images taken by the first camera; the first camera is set on a moving vehicle and continuously takes pictures, and the number of matching pixels of two adjacent first images is greater than or equal to the target number;

calculating a first mileage traveled by the vehicle after the first image was acquired last time according to the currently acquired first image and the calibration parameters of the first camera;

When the first mileage calculation fails, acquire the pulse data output by the encoder arranged on the vehicle;

calculating a second mileage traveled by the vehicle during the period when the first mileage calculation fails according to the pulse data;

According to the first mileage and the second mileage, determine the total mileage traveled by the vehicle during the target time period;

According to the total mileage, the current location of the vehicle is determined.

Optionally, the vehicle is also provided with a plurality of second cameras, and the second cameras are used to continuously photograph the side walls of the tunnel, and/or, the tunnel top arc, and/or, the tunnel ground, when determining whether the vehicle After the current positioning, also include:

acquiring a second image captured by the second camera, and identifying a target object from the second image;

When the target object is recognized, using the second image where the target object is located as the target image;

determining the location of the vehicle when the target image is captured according to the current location of the vehicle;

The location of the target image in the tunnel is determined according to the location of the vehicle when the target picture is taken.

Optionally, the currently acquired first image is a next frame image, the last acquired first image is a previous frame image, and according to the currently acquired first image, calculating The first mileage traveled by the vehicle after the last acquisition of the first image includes:

determining calibration parameters of the first camera according to a plurality of the first images and matching feature points between adjacent first images;

According to the calibration parameters of the first camera, determine the homography matrix and the distortion parameter matrix; and according to the homography matrix and the distortion parameter matrix, obtain the pixel coordinate system of the first image and the world coordinate system. conversion relationship;

Determine the pixel displacement in the image of the target pixel in the next frame of image relative to the target pixel in the previous frame of image;

According to the conversion relationship, determine that each pixel in the first image corresponds to an actual size in an actual space;

According to the actual size and the pixel displacement, the corresponding vehicle travel distance between the next frame image and the previous frame image is determined to obtain a first mileage traveled by the vehicle.

Optionally, the pulse data includes the time stamp of the output pulse and the number of pulses, and the calculation of the second mileage traveled by the vehicle during the failure of the first mileage calculation according to the pulse data includes:

determining a total number of pulses output during said first mileage calculation failure based on said timestamp and said number of pulses;

Based on the total number of pulses and the circumference of the wheel, a second mileage traveled by the vehicle during the period when the first mileage calculation fails is calculated.

Optionally, before obtaining the pulse data output by the encoder provided on the vehicle, it also includes:

When it is detected that the first abnormal situation occurs, it is determined that the calculation of the first mileage fails;

The first abnormal situation includes at least one of the following situations: the first image cannot be acquired, there is a foreign object in the target area on the first image, and the brightness of the first image is greater than or equal to a first threshold . The brightness of the first image is less than a second threshold; the number of matching pixels included in two adjacent first images is less than a target number.

Optionally, a third camera is also provided on the driving vehicle, the third camera is used to photograph the mileage recognition object, the photographing time points of the first camera and the third camera are consistent, and according to the currently acquired The first image, after calculating the first mileage traveled by the vehicle after the last acquisition of the first image, further includes:

Identifying content in a third image captured by the third camera;

If the mileage identification object is identified, correcting the first mileage according to the mileage in the mileage identification object;

After calculating the second mileage traveled by the vehicle during the period when the first mileage calculation fails according to the pulse data, further comprising:

The second mileage is corrected according to the mileage in the mileage identification object.

Optionally, after calculating the second mileage traveled by the vehicle during the period when the first mileage calculation fails according to the pulse data, the method further includes:

When the second mileage calculation fails, determine the length of the first time period corresponding to the second mileage calculation failure period;

Acquiring the mileage traveled by the vehicle in the last first time period;

The mileage traveled by the vehicle during the last first time period is taken as the second mileage traveled by the vehicle.

Optionally, after calculating the second mileage traveled by the vehicle during the period when the first mileage calculation fails according to the pulse data, the method further includes:

When the calculation of the second mileage fails, the inertial navigation data output by the inertial navigation device installed on the vehicle is obtained;

A second mileage traveled by the vehicle during the period when the calculation of the second mileage fails is calculated according to the inertial navigation data.

In the second aspect of the embodiment of the present invention, a positioning device is also provided, and the device includes:

The first image acquisition module is used to acquire a plurality of first images taken by the first camera; the first camera is arranged on a moving vehicle and takes pictures continuously, and the matching pixels of two adjacent first images The quantity is greater than or equal to the target quantity;

The first mileage calculation module is configured to calculate the first mileage traveled by the vehicle after the first image was acquired last time according to the currently acquired first image and the calibration parameters of the first camera;

A pulse data acquisition module, configured to acquire the pulse data output by the encoder provided on the vehicle when the first mileage calculation fails;

The second mileage calculation module is configured to calculate the second mileage traveled by the vehicle during the period when the first mileage calculation fails according to the pulse data;

A positioning module, configured to determine the current location of the vehicle according to the first mileage and the second mileage.

Optionally, the device also includes:

a target object recognition module, configured to acquire a second image taken by the second camera, and identify a target object from the second image;

A target image determination module, configured to use the second image where the target object is located as the target image when the target object is recognized;

a vehicle location module, configured to determine the location of the vehicle when the target image is captured according to the current location of the vehicle;

An image positioning module, configured to determine the position of the target image in the tunnel according to the position of the vehicle when the target picture is taken.

Optionally, the first mileage calculation module is further used for:

determining calibration parameters of the first camera according to a plurality of the first images and matching feature points between adjacent first images;

According to the calibration parameters of the first camera, determine the homography matrix and the distortion parameter matrix; and according to the homography matrix and the distortion parameter matrix, obtain the pixel coordinate system of the first image and the world coordinate system. conversion relationship;

Determine the pixel displacement in the image of the target pixel in the next frame of image relative to the target pixel in the previous frame of image;

According to the conversion relationship, determine that each pixel in the first image corresponds to an actual size in an actual space;

According to the actual size and the pixel displacement, the corresponding vehicle travel distance between the next frame image and the previous frame image is determined to obtain a first mileage traveled by the vehicle.

Optionally, the second mileage calculation module is further used for:

determining a total number of pulses output during said first mileage calculation failure based on said timestamp and said number of pulses;

Based on the total number of pulses and the circumference of the wheel, a second mileage traveled by the vehicle during the period when the first mileage calculation fails is calculated.

Optionally, the device also includes:

The first mileage calculation failure module is configured to determine that the first mileage calculation fails when a first abnormal situation is detected;

The first abnormal situation includes at least one of the following situations: the first image cannot be acquired, there is a foreign object in the target area on the first image, and the brightness of the first image is greater than or equal to a first threshold . The brightness of the first image is less than a second threshold; the number of matching pixels included in two adjacent first images is less than a target number.

Optionally, a third camera is also provided on the driving vehicle, the third camera is used to photograph the mileage identification object, the photographing time points of the first camera and the third camera are consistent, and the device further include:

A second identification module, configured to identify content in a third image captured by the third camera;

A first correction module, configured to correct the first mileage according to the mileage in the mileage identification object if the mileage identification object is identified;

The second correction module is configured to correct the second mileage according to the mileage in the mileage identification object.

Optionally, the device also includes:

A time length determination module, configured to determine the length of the first time period corresponding to the second mileage calculation failure period when encountering the second mileage calculation failure;

A mileage acquisition module, configured to acquire the mileage traveled by the vehicle in the last said first time period;

The second mileage determining module is configured to use the last mileage traveled by the vehicle over the length of the first time period as the second mileage traveled by the vehicle.

Optionally, the device also includes:

The inertial navigation data acquisition module is used to acquire the inertial navigation data output by the inertial navigation device installed on the vehicle when the second mileage calculation fails;

The inertial navigation mileage calculation module is configured to calculate the second mileage traveled by the vehicle during the period when the second mileage calculation fails according to the inertial navigation data.

Optionally, the positioning method also includes:

Acquiring a plurality of first images taken by the first camera; the first camera is set on a moving vehicle and continuously takes pictures, and the number of matching pixels of two adjacent first images is greater than or equal to the target number;

calculating a first mileage traveled by the vehicle after the last acquisition of the first image based on at least the currently acquired first image;

When detecting that the vehicle is slipping, acquiring inertial navigation data output by an inertial navigation device installed on the vehicle;

calculating a second mileage traveled by the vehicle during the skid period of the vehicle according to the inertial navigation data;

The current location of the vehicle is determined according to the first mileage and the second mileage.

Optionally, the vehicle is also provided with a plurality of second cameras, and the second cameras are used to continuously photograph the side walls of the tunnel, and/or, the tunnel top arc, and/or, the tunnel ground, when determining whether the vehicle After the current positioning, also include:

acquiring a second image captured by the second camera, and identifying a target object from the second image;

When the target object is recognized, using the second image where the target object is located as the target image;

determining the location of the vehicle when the target image is captured according to the current location of the vehicle;

determining the location of the target image in the tunnel according to the location of the vehicle when the target picture is taken;

Before acquiring the inertial navigation data output by the inertial navigation device arranged on the vehicle, it also includes:

Detecting whether the wheel encoder provided on the vehicle outputs pulses, and detecting whether the vehicle is displaced;

If the wheel encoder outputs pulses, it is determined that the vehicle is slipping when no displacement of the vehicle is detected;

If the wheel encoder does not output pulses, it is determined that the vehicle is slipping when a displacement of the vehicle is detected;

The method for detecting whether the vehicle is displaced includes:

determining the similarity between the first image currently acquired and the first image acquired last time;

If the similarity is greater than or equal to a first threshold, it is determined that the vehicle has not been displaced;

If the similarity is less than a first threshold, then determining that the vehicle has been displaced;

The inertial navigation data includes: the acceleration of the vehicle at the current moment and the direction and attitude angle of the vehicle, and the second mileage traveled by the vehicle is calculated according to the inertial navigation data during the skid period of the vehicle, include:

According to the first mileage calculated last time and the time period corresponding to the first mileage, determine the speed of the vehicle at the current moment;

calculating the acceleration component of the vehicle in the direction of travel according to the attitude angle and the acceleration of the vehicle;

determining a second mileage traveled by the vehicle during the vehicle skid period according to the vehicle speed at the current moment and the acceleration component, and a corresponding time period during the vehicle skid period;

The currently acquired first image is a next frame image, the last acquired first image is a previous frame image, and the last acquired image is calculated based on at least the currently acquired first image After the first image, the first mileage traveled by the vehicle includes:

determining calibration parameters of the first camera according to a plurality of the first images and matching feature points between adjacent first images;

According to the calibration parameters of the first camera, determine the homography matrix and the distortion parameter matrix; and according to the homography matrix and the distortion parameter matrix, obtain the pixel coordinate system of the first image and the world coordinate system. conversion relationship;

Determine the pixel displacement in the image of the target pixel in the next frame of image relative to the target pixel in the previous frame of image;

According to the conversion relationship, determine that each pixel in the first image corresponds to an actual size in an actual space;

According to the actual size and the pixel displacement, determine the corresponding vehicle travel distance between the next frame image and the previous frame image, and obtain the first mileage traveled by the vehicle;

The calculation of the first mileage traveled by the vehicle after the last acquisition of the first image based on at least the currently acquired first image includes:

According to the inertial navigation data output by the inertial navigation device, determine the inertial navigation mileage traveled by the vehicle after the last acquisition of the first image;

Determining the visual mileage traveled by the vehicle after the first image was acquired last time according to the matching pixel points between the currently acquired first image and the last acquired first image;

Taking the average value of the inertial mileage and the visual mileage as the first mileage traveled by the vehicle after the first image was acquired last time;

The driving vehicle is also provided with a third camera, the third camera is used to photograph the mileage identification object, the photographing time points of the first camera and the third camera are consistent, and according to the currently acquired An image, after calculating the first mileage traveled by the vehicle after the last acquisition of the first image, further includes:

Identifying content in a third image captured by the third camera;

If the mileage identification object is identified, correcting the first mileage according to the mileage in the mileage identification object;

After calculating the second mileage traveled by the vehicle during the skid period of the vehicle according to the inertial navigation data, it further includes:

The second mileage is corrected according to the mileage in the mileage identification object.

According to a third aspect of the embodiments of the present invention, a positioning device is provided, and the positioning device includes:

The first image acquisition module is used to acquire a plurality of first images taken by the first camera; the first camera is arranged on a moving vehicle and takes pictures continuously, and the matching pixels of two adjacent first images The quantity is greater than or equal to the target quantity;

The first mileage calculation module is configured to calculate the first mileage traveled by the vehicle after the last acquisition of the first image based on at least the currently acquired first image;

The inertial navigation data acquisition module is used to acquire the inertial navigation data output by the inertial navigation device installed on the vehicle when the vehicle is detected to be slipping;

A second mileage calculation module, configured to calculate a second mileage traveled by the vehicle during the skid period of the vehicle according to the inertial navigation data;

A positioning module, configured to determine the current location of the vehicle according to the first mileage and the second mileage.

According to the fourth aspect of the embodiments of the present invention, there is also provided an electronic device, including a processor and a memory, the processor and the memory are coupled, the memory is used to store computer program instructions, and when the processor executes the computer program When an instruction is issued, the electronic device is made to execute the positioning method as described in the first aspect.

According to the fifth aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, where program instructions are stored in the computer storage medium, and when the program instructions are run on the electronic device, the electronic device executes the method described in the first aspect. The positioning method described.

The present invention also provides an application publishing platform, the application publishing platform is used for publishing computer program products, wherein, when the computer program products are run on a computer, the computer is made to execute the program described in the first aspect of the embodiments of the present invention. Methods.

Combining the above-mentioned technical solutions and technical problems to be solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, closely combine the technical solution to be protected in the present invention and the results and data in the research and development process, etc., to analyze in detail how the technical solution of the present invention solves it Technical problems, some creative technical effects brought about after solving the problems. The specific description is as follows:

In the prior art, underground or mountainous tunnels and pipe corridors are not covered by the communication network, and the GPS signal is weak, so the driving equipment cannot obtain accurate positioning information for itself, detection objects or faults.

In the above-mentioned buildings, due to the complexity and variety of conditions, only one positioning method may not be able to provide high-precision positioning under all working conditions. Therefore, it is necessary to integrate multiple positioning methods and calibration methods to eliminate or reduce the reduction of positioning accuracy caused by the failure of a certain positioning method as much as possible.

Possible causes of positioning failure include: the image capture device fails to obtain an image due to occlusion, or fails to meet the positioning calculation requirements due to factors such as image quality.

The method of measuring the mileage by the circumference of the wheel (shaft) and the number of rotations of the driving equipment may affect the accuracy of positioning due to factors such as wheel (shaft) slipping and lack of air.

In order to solve the above problems, the present invention proposes a positioning method and device, and the effects brought include:

It can get rid of the limitation of communication network or GPS, integrate image, pulse signal, mileage marking, inertial navigation and other technologies, so that when one or more methods fail, they can still provide high-precision positioning information through mutual complementation and calibration.

The invention provides a method for obtaining positioning information by means of other technologies when the positioning information cannot be calculated by image analysis.

The invention provides a method for obtaining positioning information by means of other technologies when mileage measuring devices such as encoders fail.

Divide a whole tunnel into several small sections, and calibrate the location information multiple times by identifying the mileage information in the mileage mark to further improve the accuracy of positioning.

When the acceleration of the driving equipment changes drastically, the mileage information measured by the encoder can be calibrated through the inertial navigation device to further improve the accuracy of positioning.

Second, regarding the technical solution as a whole or from the perspective of a product, the technical effects and advantages of the technical solution to be protected by the present invention are specifically described as follows:

In the present invention, a plurality of first images taken by the first camera are obtained; the first camera is set on a moving vehicle and takes pictures continuously, and the number of matching pixels of two adjacent first images is greater than or Equal to the number of targets; according to the currently acquired first image and the calibration parameters of the first camera, calculate the first mileage traveled by the vehicle after the last acquisition of the first image; when encountering the first When the mileage calculation fails, the pulse data output by the encoder installed on the vehicle is obtained; according to the pulse data, the second mileage traveled by the vehicle during the failure of the first mileage calculation is calculated; according to the The first mileage and the second mileage determine the current location of the vehicle. In this scheme, there is no need to use satellite signals or network signals, and the first image and encoder can be used to achieve high-precision positioning in the tunnel, and a method for using the encoder to locate when the first mileage calculation fails is given. The success rate of positioning is improved.

Thirdly, as an auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in that the present invention does not rely on infrastructure such as GPS or communication networks, and realizes high-precision positioning in tunnels, filling the gap in this field at home and abroad.

The invention solves the urgent problem that the driving equipment cannot provide high-precision positioning information in response to the diversity and complexity of the working environment in the above-mentioned buildings. Eliminate or reduce the reduction in positioning accuracy caused by the failure of a certain positioning method by fusing technical methods and calibration methods such as images, pulse signals, mileage markers, and inertial navigation. It provides practical and effective methods and device descriptions for driving equipment to achieve high-precision positioning in tunnels, pipe corridors and other buildings that are not covered by communication networks such as underground or mountainous areas, and solves the current problems in tunnel construction, inspection, maintenance, repair and other operations. The urgent problem of inaccurate positioning.

Description of drawings

FIG. 1 is a schematic diagram of a tunnel defect detection system provided by an embodiment of the present invention;

FIG. 2 is a flow chart of the steps of the first positioning method provided by the embodiment of the present invention;

FIG. 3 is a flow chart of the steps of the second positioning method provided by the embodiment of the present invention;

FIG. 4 is a structural block diagram of a positioning device provided by an embodiment of the present invention;

FIG. 5 is a structural block diagram of an electronic device provided by an embodiment of the present invention;

FIG. 6 is a flow chart of the steps of the first positioning method provided by the embodiment of the present invention;

FIG. 7 is a flow chart of the steps of the second positioning method provided by the embodiment of the present invention;

FIG. 8 is a structural block diagram of a positioning device provided by an embodiment of the present invention;

In the figure: 1. Third camera; 2. First camera; 3. First light source array; 4. Second light source array; 5. Industrial computer; 6. Central control unit; 7. Encoder; 8. Inertial navigation device .

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present invention, the following will describe the technical solution in the embodiment of the present invention in conjunction with the accompanying drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the present invention Examples, but not all examples. All embodiments based on the present invention shall belong to the protection scope of the present invention.

1. Explain the embodiment. In order to make those skilled in the art fully understand how to implement the present invention, this part is an explanatory embodiment for explaining the technical solution of the claims.

Example 1

The vehicle in the embodiment of the present invention is a refitted vehicle equipped with a camera and a light source array, and the vehicle may also include an encoder, an industrial computer and a central control unit. The encoder is connected to the driven wheel shaft through an adapter ring to detect the speed and angle of the wheel movement, and to count the number of revolutions. The industrial computer and the central control unit are fixed in the compartment.

There is acquisition and processing software on the industrial computer, which is used to acquire the image frames sent by the camera, process the image frames, and calculate the mileage data of the vehicle movement. The central control unit can read the mileage data transmitted by the industrial computer and update the mileage data. The central control unit is also responsible for the power supply and trigger drive of the camera and light source array. The camera and the light source array can be driven by an external hard trigger, or by setting a driving chip inside and driving them internally, which is not limited in the present invention. The above vehicle, camera, light source array, encoder, industrial computer, and central control unit form a tunnel defect detection system. Specifically, the execution subject of the embodiment of the present invention may be an industrial computer and a central control unit.

The aforementioned encoders, inertial navigation devices and part of the positioning cameras all play the role of mileage positioning, which can be increased or decreased according to actual needs, as long as the mileage positioning can be realized. The tunnel defect detection system may include at least one of an encoder, an inertial navigation device, and a positioning camera.

As shown in FIG. 1 , it is a schematic diagram of a tunnel defect detection system provided by an embodiment of the present invention. Referring to FIG. 1 , the tunnel defect detection system includes a third camera 1 , a first camera 2 , a first light source array 3 , a second light source array 4 , an industrial computer 5 , a central control unit 6 , and an encoder 7 .

Wherein, the third camera 1 may be a camera array with a plurality of cameras facing different directions, and is used for collecting images of the tunnel lining surface in all directions and in a large range. The first light source array 3 includes a plurality of flashlights, which can be used to assist the third camera 1 to capture images with brightness and clarity that meet requirements. The first camera 2 and the second light source array 4 can be installed and fixed on the vehicle frame to take pictures of the track bed between the two rails on the tunnel floor.

The industrial computer 5 and the central control unit 6 are fixed in the compartment. The industrial computer 5 can receive the image data sent by the third camera 1 and the first camera 2, process the image data, output the mileage data, and transmit it to the central control unit 6 through the serial port. The central control unit 6 can calibrate the mileage data. The encoder 7 is connected with the driven wheel of the vehicle through an adapter ring.

From the time the vehicle starts to drive, the third camera 1 and the first camera 2 continuously take pictures in response to the pulse signal, and the encoder 7 and the inertial navigation device 8 can also be always on, and continue to work as the vehicle drives, and continuously output relevant information. The pulse data and inertial navigation data are sent to the industrial computer 5 .

The aforementioned vehicles can be replaced by drones.

As shown in FIG. 2 , it is a flow chart of the steps of the first positioning method provided by the embodiment of the present invention. The methods include:

101. Acquire multiple first images taken by the first camera; the first camera is installed on a moving vehicle and takes pictures continuously, and the number of matching pixels of two adjacent first images is greater than or equal to the target number .

The first camera is set on a moving vehicle, and is periodically or irregularly triggered externally or internally to take pictures. For example, the central control unit 6 generates trigger pulses for the first camera 2 and the second light source array 4 every second. give light.

It can be understood that the vehicle in the embodiment of the present invention can be replaced with other vehicles, such as unmanned aerial vehicles, flatbed trucks, etc., as long as the purpose of the present invention can be achieved, and there is no limitation here.

The shooting area of the first camera should meet the requirement that the captured images have continuity, so that it is convenient to perform feature matching on images of adjacent frames. For example, the camera orientation of the first camera may be set to be the track area at the bottom of the train.

The first camera performs continuous shooting according to the trigger signal to obtain multiple first images. In order to use the first image to calculate the mileage traveled by the vehicle, it is necessary to perform feature point extraction on the first image, and to perform feature point matching on images of adjacent frames.

Time stamps may be included in the first image to facilitate identification of adjacent frame images.

The feature point matching of two adjacent images can be realized by using the homography matrix. Because the homography matrix has 8 degrees of freedom, at least 4 pairs of corresponding points are required to calculate the homography matrix. Therefore, two adjacent first images need to have matching pixel points greater than or equal to the target number, and the target number may be four.

In real application scenarios, the point pairs calculated by the present invention will contain noise. For example, the position of the point deviates by a few pixels, and even the phenomenon of feature point pair mismatch occurs. If only 4 point pairs are used to calculate the homography matrix, there will be a large error. Therefore, in order to make the calculation more accurate, generally more than 4 point pairs are used to calculate the homography matrix.

In addition, in order to make each pair of adjacent first images include a target number of matching pixels, it is necessary to precisely calculate the shooting interval of the first camera. In order to make the number of matching pixels in the first images uniform, the vehicle preferably drives at a constant speed.

102. According to the first image currently acquired and the calibration parameters of the first camera, calculate a first mileage traveled by the vehicle after the first image was acquired last time.

The currently acquired first image and the last acquired first image are two adjacent frames of images. By matching the feature points of the two frames of images, the position changes of the pixels in the images can be obtained. How to map the position change scale of the pixels in the image to the actual mileage of the vehicle during the time period when the two images were taken in reality? Coordinate mapping between the pixel coordinate system and the world coordinate system is required here.

By calibrating the camera, the external parameter matrix, internal parameter matrix, and distortion matrix of the first camera can be obtained. According to the calibration parameters of the first camera, the present invention can realize the coordinate mapping between the pixel coordinate system and the world coordinate system, that is, accurately realize The projection process from the point in reality to the corresponding pixel in the image.

In this way, with the help of the calibration parameters of the first camera, the first mileage actually traveled by the vehicle can be obtained through the position changes of the pixels in the first image.

The above-mentioned scale mapping is performed on each pair of adjacent first images to obtain a plurality of first mileages, and then the total mileage of the vehicle within the corresponding time period from the initial frame to the current frame can be obtained.

Optionally, step 102 includes the following steps A1-A5:

A1. Determine the calibration parameters of the first camera according to the multiple first images and matching feature points between adjacent first images.

The calibration parameters of the first camera are determined by calibrating the camera. The camera calibration parameters can establish the mapping relationship between the spatial three-dimensional coordinate system and the camera image coordinate system.

Camera calibration parameters are usually divided into internal parameter matrix, external parameter matrix, and distortion parameter matrix.

Specifically, the calibration parameters of the first camera can be calculated through the following steps A11-A15:

A11. Traverse the m first images, detect corner points in the first images, and obtain n corner points. Solve the equation AHj=0 (a total of 2n equations and 9 unknowns) to obtain the initial value of the homography matrix Hj, and consider the influence of noise to perform nonlinear optimization.

A12, solution of internal parameters of the camera: synthesize the homography matrix Hj of the m first image, solve the equation vb=0 (2m equations in total, 6 unknowns), decompose the intermediate matrix b, and obtain the internal parameter matrix K (fx, fy, u0, v0, γ5 parameters).

A13. Solve the external parameters of the camera: calculate Rj and tj.

A14. Solving distortion coefficients: integrating the n corner points in the m first images, solving the equation Dk=d (a total of 2mn equations, 2 unknowns), and obtaining the distortion parameters and the distortion parameter matrix.

A15. Parameter optimization: through nonlinear least square method, optimize K, Rj and tj and K1, K2 to obtain the internal and external parameters of the camera.

A2. Determine the homography matrix and the distortion parameter matrix according to the calibration parameters of the first camera; and obtain the pixel coordinate system and the world coordinate system of the first image according to the homography matrix and the distortion parameter matrix. conversion relationship between them.

The homography transformation is used to describe the position mapping relationship between the object in the world coordinate system and the pixel coordinate system, and the corresponding transformation matrix is called the homography matrix. The homography matrix H is defined as:

Among them, M is the internal reference matrix of the camera, and M is defined as follows:

Among them, r and t are the extrinsic parameter matrix of the camera.

In step A1, the extrinsic and internal reference matrices of the first camera have been calculated and substituted into the homography matrix H to obtain a homography matrix describing the conversion relationship between the pixel coordinate system and the world coordinate system of the first image.

In geometric optics and cathode ray tube displays, distortion is a shift from a rectilinear projection. When determining the transformation relationship, it is necessary to consider the offset of objects in the real world projected onto the image. Therefore, to perform coordinate transformation, in addition to the homography matrix, the distortion parameter matrix is also required.

Specifically, the pixel coordinate system of the first image is converted to the world coordinate system by using the homography matrix, and then the transformed coordinates are corrected by using the distortion parameter matrix.

A3. Determine the pixel displacement in the image of the target pixel in the next frame of image relative to the target pixel in the previous frame of image.

Select any pair of matching pixel points from the two adjacent first images to obtain a pair of target pixel points. Determine the coordinate position of the target pixel point in the two first images respectively, according to the change of the coordinate position, the target pixel point in the first image of the next frame can be obtained, relative to the target pixel point in the previous frame image, Pixel displacement in the image.

Pixel displacement refers to the number of pixels passed by a pixel from the starting position to the target position.

A4. According to the conversion relationship, determine that each pixel in the first image corresponds to an actual size in an actual space.

According to the conversion relationship, the pixels in the first image can be projected from the image coordinate system to the world coordinate system, and in the world coordinate system, the actual size of each pixel corresponding to the actual space can be obtained.

A5. According to the actual size and the pixel displacement, determine the corresponding vehicle travel distance between the next frame image and the previous frame image, and obtain the first mileage traveled by the vehicle.

By mapping the pixel displacement of the target pixel point to the world coordinate system, the corresponding vehicle displacement between the next frame image and the previous frame image can be obtained, that is, the vehicle travel distance. Specifically, the product of the pixel displacement and the actual size can be calculated to obtain the vehicle displacement, that is, the first mileage.

In the above steps A1-A5, the actual size of each pixel is determined according to the conversion relationship between the pixel coordinate system and the world coordinate system, and the actual size is combined with the pixel displacement in the first image to obtain the pixel displacement corresponding to the actual space The displacement in , and then get the first mileage. This calculation method is not limited by satellite positioning signals or network signals, and the accuracy of the calculation results is not affected by the speed and state of the vehicle. The first mileage is calculated using the mapping relationship between pixels and the actual space, and there is almost no error. Higher precision.

103. Acquire pulse data output by an encoder installed on the vehicle when the first mileage calculation fails.

The adjacent first image needs to be used for the calculation of the first mileage. If the industrial computer does not acquire the first image, or if the acquired first image is abnormal, the first mileage cannot be calculated, that is, the calculation of the first mileage fails; Or, the hardware failure or software error of the industrial computer may also cause the first mileage calculation to fail.

If the calculation of the first mileage is successful, the program outputs a specific value; if the calculation of the first mileage fails, the program outputs a failure code, so it can be determined that the calculation of the first mileage fails.

When the calculation of the first mileage fails, in order to maintain the availability of the tunnel defect detection system, data can be obtained from other ways to calculate the first mileage. For example, to obtain pulse data from an encoder.

The encoder is connected to the vehicle tires to detect the speed and angle of the wheel rotation, and calculate the number of rotations of the wheel, etc. The encoder sends out a pulse signal every time the wheel rotates one revolution. Multiple pulse signals make up pulse data.

104. According to the pulse data, calculate the second mileage traveled by the vehicle during the period when the calculation of the first mileage fails.

The pulse data of the encoder can be used to calculate the number of revolutions of the wheel within a period of time, and then obtain the circumference of the wheel, then the product of the number of revolutions and the circumference is the mileage of the vehicle within a period of time.

If the program continues to output failure codes, the first mileage calculation fails during this period. During the failure of the first mileage calculation, the encoder is used to calculate the second mileage within the time period.

Optionally, the pulse data includes the time stamp of the output pulse and the number of pulses, and the step 104 includes the following steps B1-B2:

B1. According to the time stamp and the number of pulses, determine the total number of pulses output during the first mileage calculation failure period.

Get the start time point when the first mileage calculation failure code is received for the first time, and the end time point when the failure code is received last time, the time period between the start time point and the end time point is the first mileage calculation failure period.

According to the time stamp in the pulse data, the number of pulses received in the time period is obtained to obtain the total number of pulses.

B2. According to the total number of pulses and the circumference of the wheel, calculate a second mileage traveled by the vehicle during the period when the first mileage calculation fails.

The total number of pulses is multiplied by the circumference of the wheel to obtain the second mileage traveled by the vehicle during the failure of the first mileage calculation.

105. Determine the current location of the vehicle according to the first mileage and the second mileage.

Before the current time point, there may be a period of time in which the calculation of the first mileage is successful, and a period of time in which the calculation of the first mileage fails, and there will be multiple first mileages and second mileages. The total mileage of the current vehicle can be obtained by calculating the sum of all the first mileage and the second mileage.

Obtain the initial location of the vehicle when the first image is taken; starting from the initial location, add the displacement of the total mileage in the forward direction of the vehicle to obtain the current location of the vehicle.

When calculating the location of the vehicle next time, you only need to add the corresponding total mileage to the previous location. The location of each moment during the driving of the vehicle can be calculated by the above method.

To sum up, in the embodiment of the present invention, a plurality of first images taken by the first camera are obtained; the first camera is set on a driving vehicle and continuously takes pictures, and the matching of two adjacent first images The number of pixels is greater than or equal to the target number; according to the currently acquired first image and the calibration parameters of the first camera, calculate the first mileage traveled by the vehicle after the last acquisition of the first image; When the calculation of the first mileage fails, the pulse data output by the encoder provided on the vehicle is obtained; according to the pulse data, the second mileage of the vehicle during the period of the failure of the calculation of the first mileage is calculated. Mileage: Determine the current location of the vehicle according to the first mileage and the second mileage. In this scheme, there is no need to use satellite signals or network signals, and the first image and encoder can be used to achieve high-precision positioning in the tunnel, and a method for using the encoder to locate when the first mileage calculation fails is given. The success rate of positioning is improved.

FIG. 3 is a flowchart of steps of a second positioning method provided by an embodiment of the present invention.

The methods include:

201. Acquire multiple first images taken by the first camera; the first camera is installed on a moving vehicle and takes pictures continuously, and the number of matching pixels of two adjacent first images is greater than or equal to the target number .

In the embodiment of the present invention, step 201 may refer to step 101, which will not be repeated here.

202. According to the currently acquired first image, calculate a first mileage traveled by the vehicle after the last acquisition of the first image.

In the embodiment of the present invention, step 202 may refer to step 102, which will not be repeated here.

203. The driving vehicle is also provided with a third camera, the third camera is used to photograph the mileage recognition object, and the photographing time points of the first camera and the third camera are consistent; for the third camera The content in the captured third image is identified.

Due to the inevitable noise information in the first image, the calculation of the first mileage will generate errors, and the errors will be accumulated many times, resulting in large deviations. The encoder calculates the second mileage by accumulative counting, and the error of the second mileage is also relatively large due to problems such as wheel slipping and idling that may occur. In order to reduce the error of mileage calculation, a mechanism of fixed distance calibration is introduced. Calibrate the calculated first and second mileages by using mileage recognition objects that are commonly set up in tunnels, such as road signs, distance reminder signs, etc.

A third camera needs to be introduced to shoot at the mileage identification object in the tunnel, so as to use the mileage data in the captured third image to calibrate the calculated mileage.

Optionally, the mileage recognition object includes one of milestones on the side of the tunnel and milestones on the tunnel floor.

Milestones in the tunnel may be located on both sides of the road on the side of the tunnel, and may be upright road piles, or mileage text sprayed on the wall, or may be mileage text sprayed on the tunnel ground.

The first camera and the third camera use the same trigger signal, so that the shooting time of the two cameras can be consistent. The camera head of the third camera can be aimed at any of the above positions for continuous shooting. In the plurality of third images captured by the third camera, whether a milestone is present is identified.

204. If the mileage identification object is identified, correct the first mileage according to the mileage in the mileage identification object.

Continuously identify a plurality of third images, and if it is recognized that the current third image includes a mileage identification object, the currently calculated first mileage is corrected by using the mileage in the mileage identification object.

For example, if the currently calculated first mileage value is 99.9m, and the currently acquired third image contains a mileage recognition object with a mileage of 100m, the first mileage is corrected to 100m.

Every time the mileage recognition object is recognized in the third image, the mileage number therein can be used to correct the current first mileage, so that the first mileage can maintain the actual accuracy, thereby ensuring the accuracy of positioning.

205. When a first abnormal situation is detected, determine that the calculation of the first mileage fails; the first abnormal situation includes at least one of the following situations: the first image, the first There is a foreign object in the target area on the image, the brightness of the first image is greater than or equal to the first threshold, and the brightness of the first image is smaller than the second threshold; the number of matching pixels included in two adjacent first images less than the target amount.

The reasons for the failure of the first mileage calculation include two aspects: first, the first image is not obtained; second, the first image is obtained, but the first image does not meet the requirements.

There may be two reasons for not getting the first image: one is related to the first camera, such as the failure of the first camera and unable to take pictures, the battery is exhausted, and the memory card of the first camera is full, etc.; the other is the receiving The reasons related to the IPC of the first image, such as the failure of the IPC and the inability to receive the first image, the memory card of the IPC is full, etc.

The situations where the first image does not meet the requirements mainly include: there is a foreign object in the target area on the first image, the brightness of the first image is greater than or equal to the first threshold, and the brightness of the first image is smaller than the second threshold; The number of matching pixels included in the two adjacent first images is less than the target number.

Because there is often water in the tunnel, or there are flying insects and birds in the air, when these things are near the camera, it will cause foreign objects to appear in the first image. If the foreign objects appear in the matching pixel area, it will cause two adjacent The pixel point matching of the first image fails; while the brightness does not meet the requirements, it is generally caused by underexposure or overexposure of the flash, or insufficient exposure time of the flash. All of these may cause the number of matching pixels included in the two adjacent first images to be less than the target number, resulting in failure to calculate the first mileage.

In addition, when there is a culvert on the side wall of the tunnel, or there is a platform area in the subway tunnel, if the camera of the first camera is aimed at the tunnel wall, when the vehicle passes through these areas, there will be no definite subject in the camera, which will cause the first The image is blurred or the image is empty. These situations will also cause the number of matching pixels included in the two adjacent first images to be less than the target number.

The first image will appear blurry,

206. Acquire pulse data output by an encoder installed on the vehicle when the first mileage calculation fails.

In the embodiment of the present invention, step 206 may refer to step 103, which will not be repeated here.

207. According to the pulse data, calculate the second mileage traveled by the vehicle during the period when the calculation of the first mileage fails.

In the embodiment of the present invention, step 207 may refer to step 104, which will not be repeated here.

208. Correct the second mileage according to the mileage in the mileage identification object.

Similarly, the second mileage can also be corrected by using the mileage number in the mileage identification object, which is similar to the process of correcting the first mileage.

After step 208, when the second mileage calculation fails, steps 209-211 may be performed; when an inertial navigation device is installed on the vehicle and the second mileage calculation fails, steps 212-213 may be performed.

209. When the second mileage calculation fails, determine the length of the first time period corresponding to the second mileage calculation failure period.

210. Obtain the mileage traveled by the vehicle during the last first time period.

211. Use the mileage traveled by the vehicle during the last first time period as the second mileage traveled by the vehicle.

In step 209-step 211, the solution when the second mileage calculation fails is given. The second mileage is estimated according to the mileage traveled by the vehicle in the previous first time period. Because the mileage of the vehicle generally does not change much in a period of time, the feasibility of this estimation is better.

212. Acquire the inertial navigation data output by the inertial navigation device installed on the vehicle when the second mileage calculation fails.

213. Calculate the second mileage traveled by the vehicle during the period when the calculation of the second mileage fails according to the inertial navigation data.

In step 212-step 213, another solution is given when the second mileage calculation fails. In the case that the inertial navigation device is pre-installed on the vehicle, the second mileage can be calculated using the inertial navigation data.

Referring to FIG. 2, an inertial navigation device 8 is installed on the floor of a vehicle test stand to measure the acceleration and angular velocity of the vehicle.

When the vehicle skids, the encoder fails. At this time, the inertial navigation device can effectively measure the acceleration of the vehicle due to the skid, and then calculate the driving distance when the vehicle skids. Compared with the method in steps 209-211, this method has higher positioning accuracy.

214. Determine the current location of the vehicle according to the first mileage and the second mileage.

In the embodiment of the present invention, step 214 may refer to step 105, which will not be repeated here.

215. The vehicle is also provided with a plurality of second cameras, and the second cameras are used to continuously photograph the side walls of the tunnel, and/or the top arc of the tunnel, and/or the ground of the tunnel; A second image of , and identify the target object from the second image.

The second camera can be a camera array with multiple cameras facing in different directions, and is used to collect images of tunnel lining sections in all directions and in a large range. Specifically, the second camera may be used to photograph the side wall of the tunnel, and/or the top arc of the tunnel, and/or the ground of the tunnel.

The industrial computer recognizes the target object from the second image taken by the second camera, and the target object is the lesion on the surface of the tunnel.

Diseases on the tunnel surface may include lining cracks, water leakage, freezing damage, lining spalling, tunnel deformation, pavement damage, etc. By means of an image recognition algorithm, the aforementioned lesion can be identified in the second image.

216. When the target object is recognized, use the second image where the target object is located as a target image.

When a disease is identified, the second image containing the disease is used as the target image.

217. According to the current location of the vehicle, determine the location of the vehicle when the target image is captured.

According to the current location of the vehicle, the location of the vehicle at each moment can be obtained. Therefore, the location of the vehicle when shooting the target object is also easy to obtain.

218. Determine the location of the target image in the tunnel according to the location of the vehicle when the target picture is taken.

The location of the disease in the tunnel can be realized by locating the target image. By determining the location of the vehicle when the target image is captured, the location of the target image in the tunnel can be roughly determined.

To sum up, in the embodiment of the present invention, a solution when the second mileage calculation fails is given. That is, the second mileage is estimated based on the mileage traveled by the vehicle in the previous first time period. Because the mileage of the vehicle generally does not change much in a period of time, the feasibility of this estimation is better, and it does not need to rely on other hardware equipment; in addition, when the inertial navigation device is pre-installed on the vehicle, the inertial navigation data can be used to calculate the second mileage. When the vehicle skids, the encoder fails. At this time, the inertial navigation device can effectively measure the acceleration of the vehicle due to the skid, and then calculate the driving distance when the vehicle skids. This method has higher positioning accuracy.

Fig. 4 is a positioning device provided by an embodiment of the present invention. The positioning device is set on an electronic device, and in the embodiment of the present invention, the electronic device is the industrial computer shown in FIG. 1 . As shown in Figure 4, the device 300 includes:

The first image acquisition module 301 is configured to acquire a plurality of first images taken by the first camera; the first camera is arranged on a moving vehicle and continuously takes pictures, and the matching pixels of two adjacent first images The number of points is greater than or equal to the target amount;

The first mileage calculation module 302 is configured to calculate the first mileage traveled by the vehicle after the first image was acquired last time according to the currently acquired first image and the calibration parameters of the first camera;

A pulse data acquisition module 303, configured to acquire the pulse data output by the encoder provided on the vehicle when the first mileage calculation fails;

The second mileage calculation module 304 is configured to calculate the second mileage traveled by the vehicle during the period when the first mileage calculation fails according to the pulse data;

The location module 305 is configured to determine the current location of the vehicle according to the first mileage and the second mileage.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

Fig. 5 is a structural block diagram of an electronic device 400 provided by an embodiment of the present invention. For example, the electronic device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.

5, electronic device 400 may include one or more of the following components: processing component 402, memory 404, power component 406, multimedia component 408, audio component 410, input/output (I/O) interface 412, sensor component 414 , and the communication component 416 .

The processing component 402 generally controls the overall operations of the electronic device 400, such as those associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 402 may include one or more modules that facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402 .

Memory 404 is configured to store various types of data to support operations at device 400 . Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phonebook data, messages, pictures, videos, and the like. The memory 404 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

The power supply component 406 provides power to various components of the electronic device 400 . Power components 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 400 .

The multimedia component 408 includes a screen providing an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect a duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 408 includes a front camera and/or a rear camera. When the device 400 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 410 is configured to output and/or input audio signals. For example, the audio component 410 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 400 is in operation modes, such as call mode, recording mode and voice recognition mode. Received audio signals may be further stored in memory 404 or sent via communication component 416 . In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

The I/O interface 412 provides an interface between the processing component 402 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

Sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for electronic device 400 . For example, the sensor component 414 can detect the open/closed state of the device 400, the relative positioning of components, such as the display and the keypad of the electronic device 400, the sensor component 414 can also detect the electronic device 400 or a component of the electronic device 400 The position change of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration/deceleration of the electronic device 400 and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 414 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 416 is configured to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 4G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 416 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

Example 2

The vehicle in the embodiment of the present invention is a refitted vehicle equipped with a camera and a light source array, and the vehicle may also include an encoder, an inertial navigation device, an industrial computer and a central control unit. The encoder is connected to the driven wheel shaft through an adapter ring to detect the speed and angle of the wheel movement, and to count the number of revolutions. The industrial computer and the central control unit are fixed in the compartment.

There is acquisition and processing software on the industrial computer, which is used to acquire the image frames sent by the camera, process the image frames, and calculate the mileage data of the vehicle movement. The execution subject of the embodiment of the present invention may be an industrial computer. The central control unit can read the mileage data transmitted by the industrial computer and update the mileage data. The central control unit is also responsible for the power supply and trigger drive of the camera and light source array. The camera and the light source array can be driven by an external hard trigger, or by setting a driving chip inside and driving them internally, which is not limited in the present invention. The above vehicle, camera, light source array, encoder, inertial navigation device, industrial computer, and central control unit form a tunnel defect detection system.

The aforementioned encoders, inertial navigation devices and part of the positioning cameras all play the role of mileage positioning, which can be increased or decreased according to actual needs, as long as the mileage positioning can be realized. The tunnel defect detection system may include at least one of an encoder, an inertial navigation device, and a positioning camera.

As shown in FIG. 1 , it is a schematic diagram of a tunnel defect detection system provided by an embodiment of the present invention. Referring to FIG. 1 , the tunnel defect detection system includes a third camera 1 , a first camera 2 , a first light source array 3 , a second light source array 4 , an industrial computer 5 , a central control unit 6 , an encoder 7 , and an inertial navigation device 8 .

Wherein, the third camera 1 may be a camera array with a plurality of cameras facing in different directions, and is used for collecting tunnel lining section images in all directions and in a large range. The first light source array 3 includes a plurality of flashlights, which can be used to assist the third camera 1 to capture images with brightness and clarity that meet requirements. The first camera 2 and the second light source array 4 can be installed and fixed on the vehicle frame to take pictures of the track bed between the two rails on the tunnel floor.

The industrial computer 5 and the central control unit 6 are fixed in the compartment. The encoder 7 is connected with the driven wheel of the vehicle through an adapter ring. The inertial navigation device 8 is installed on the floor of the vehicle test bench through anti-vibration devices, and is used to measure the acceleration and angular velocity of the vehicle.

From the time when the vehicle is running, the third camera 1 and the first camera 2 continuously take pictures in response to the pulse signal, and the encoder 7 and the inertial navigation device 8 can also be always on, and continue to work as the vehicle is running, and continuously output relevant pulses The data and the inertial navigation data are sent to the industrial computer 5 .

The industrial computer 5 can receive the image data sent by the third camera 1 and the first camera 2, and the data sent by the encoder 7 and the inertial navigation device 8, and the industrial computer 5 processes the received data, outputs the mileage data, and transmits them through the serial port to the central control unit 6. The central control unit 6 can calibrate the mileage data.

The aforementioned vehicles can be replaced by drones.

As shown in FIG. 6 , it is a flow chart of the steps of the first positioning method provided by the embodiment of the present invention. The methods include:

601. Acquire multiple first images taken by the first camera; the first camera is installed on a moving vehicle and takes pictures continuously, and the number of matching pixels in two adjacent first images is greater than or equal to the target number .

The first camera is set on a moving vehicle, and is periodically or irregularly triggered externally or internally to take pictures. For example, the central control unit 6 generates trigger pulses for the first camera 2 and the second light source array 4 every second. give light.

It can be understood that the vehicle in the embodiment of the present invention can be replaced with other vehicles, such as unmanned aerial vehicles, flatbed trucks, etc., as long as the purpose of the present invention can be achieved, and there is no limitation here.

The shooting area of the first camera should meet the requirement that the captured images have continuity, so that it is convenient to perform feature matching on images of adjacent frames. For example, the camera orientation of the first camera may be set to be the track area at the bottom of the train.

The first camera performs continuous shooting according to the trigger signal to obtain multiple first images. In order to use the first image to calculate the mileage traveled by the vehicle, it is necessary to perform feature point extraction on the first image, and to perform feature point matching on images of adjacent frames.

Time stamps may be included in the first image to facilitate identification of adjacent frame images.

The feature point matching of two adjacent images can be realized by using the homography matrix. Because the homography matrix has 8 degrees of freedom, at least 4 pairs of corresponding points are required to calculate the homography matrix. Therefore, two adjacent first images need to have matching pixel points greater than or equal to the target number, and the target number may be four.

In real application scenarios, the point pairs calculated by the present invention will contain noise. For example, the position of the point deviates by a few pixels, and even the feature point pair mismatch phenomenon occurs. If only 4 point pairs are used to calculate the homography matrix, there will be a large error. Therefore, in order to make the calculation more accurate, generally more than 4 point pairs are used to calculate the homography matrix.

In addition, in order to make each pair of adjacent first images include a target number of matching pixels, it is necessary to precisely calculate the shooting interval of the first camera. In order to make the number of matching pixels in the first images uniform, the vehicle preferably drives at a constant speed.

602. Calculate a first mileage traveled by the vehicle after the first image was acquired last time according to at least the currently acquired first image.

The currently acquired first image and the last acquired first image are two adjacent frames of images. By matching the feature points of the two frames of images, the position changes of the pixels in the images can be obtained. How to map the position change scale of the pixels in the image to the actual mileage of the vehicle during the time period when the two images were taken in reality? Coordinate mapping between the pixel coordinate system and the world coordinate system is required here.

By calibrating the camera, the external parameter matrix, internal parameter matrix, and distortion matrix of the first camera can be obtained. According to the calibration parameters of the first camera, the present invention can realize the coordinate mapping between the pixel coordinate system and the world coordinate system, that is, accurately realize The projection process from the point in reality to the corresponding pixel in the image.

In this way, with the help of the calibration parameters of the first camera, the first mileage actually traveled by the vehicle can be obtained through the position changes of the pixels in the first image.

The above-mentioned scale mapping is performed on each pair of adjacent first images to obtain a plurality of first mileages, and then the total mileage of the vehicle within the corresponding time period from the initial frame to the current frame can be obtained.

In addition, if the vehicle is traveling at a constant speed, the mileage of the vehicle in the same time period is the same. That is: the first mileage traveled by the vehicle during the time period when the last pair of adjacent first images was taken can be taken as the first mileage traveled by the vehicle during the time period corresponding to the pair of adjacent first images currently taken .

If the vehicle is not traveling at a constant speed, an acceleration sensor can be introduced to measure the acceleration of the vehicle. According to the instantaneous speed and acceleration of the vehicle when the last frame of the first image was captured, and the time difference between the capture of the previous frame of the first image and the current first image, calculate the driving time of the vehicle after the last acquisition of the first image first mile.

Optionally, step 102 may include the following steps A1-step A5, or, step 102 may include the following steps B1-step B5.

Wherein, step A1-step A5 is to calculate the first mileage traveled by the vehicle after the last acquisition of the first image according to the currently acquired first image and the calibration parameters of the first camera.

Step B1-Step B3 is to calculate the inertial mileage based on the currently acquired inertial navigation data, calculate the visual mileage based on the first image and the calibration parameters of the camera, and finally determine the first mileage based on the average value of the two.

A1. Determine the calibration parameters of the first camera according to the multiple first images and matching feature points between adjacent first images.

The calibration parameters of the first camera are determined by calibrating the camera. The camera calibration parameters can establish the mapping relationship between the spatial three-dimensional coordinate system and the camera image coordinate system.

Camera calibration parameters are usually divided into internal parameter matrix, external parameter matrix, and distortion parameter matrix.

Specifically, the calibration parameters of the first camera can be calculated through the following steps A11-A15:

A11. Traverse the m first images, detect corner points in the first images, and obtain n corner points. Solve the equation AHj=0 (a total of 2n equations, 9 unknowns), get the initial value of the homography matrix Hj, and consider the influence of noise, conduct nonlinear optimization.

A12, camera internal parameter solution: synthesize the homography matrix Hj of m first images, solve the equation vb=0 (2m equations in total, 6 unknowns), decompose the intermediate matrix b, and obtain the internal parameter matrix K(fx , fy, u0, v0, γ5 parameters).

A13. Solve the external parameters of the camera: calculate Rj and tj.

A14. Solving distortion coefficients: integrating the n corner points in the m first images, solving the equation Dk=d (a total of 2mn equations, 2 unknowns), and obtaining the distortion parameters and the distortion parameter matrix.

A15. Parameter optimization: through nonlinear least square method, optimize K, Rj and tj and K1, K2 to obtain the internal and external parameters of the camera.

A2. Determine the homography matrix and the distortion parameter matrix according to the calibration parameters of the first camera; and obtain the pixel coordinate system and the world coordinate system of the first image according to the homography matrix and the distortion parameter matrix. conversion relationship between them.

The homography transformation is used to describe the position mapping relationship between the object in the world coordinate system and the pixel coordinate system, and the corresponding transformation matrix is called the homography matrix. The homography matrix H is defined as:

Among them, M is the internal reference matrix of the camera, and M is defined as follows:

Among them, r and t are the extrinsic parameter matrix of the camera.

In step A1, the extrinsic and internal reference matrices of the first camera have been calculated and substituted into the homography matrix H to obtain a homography matrix describing the conversion relationship between the pixel coordinate system and the world coordinate system of the first image.

In geometric optics and cathode ray tube displays, distortion is a shift from a rectilinear projection. When determining the transformation relationship, it is necessary to consider the offset of objects in the real world projected onto the image. Therefore, to perform coordinate transformation, in addition to the homography matrix, the distortion parameter matrix is also required.

Specifically, the pixel coordinate system of the first image is converted to the world coordinate system by using the homography matrix, and then the transformed coordinates are corrected by using the distortion parameter matrix.

A3. Determine the pixel displacement in the image of the target pixel in the next frame of image relative to the target pixel in the previous frame of image.

Select any pair of matching pixel points from the two adjacent first images to obtain a pair of target pixel points. Determine the coordinate position of the target pixel point in the two first images respectively, according to the change of the coordinate position, the target pixel point in the first image of the next frame can be obtained, relative to the target pixel point in the previous frame image, Pixel displacement in the image.

Pixel displacement refers to the number of pixels passed by a pixel from the starting position to the target position.

A4. According to the conversion relationship, determine that each pixel in the first image corresponds to an actual size in an actual space.

According to the conversion relationship, the pixels in the first image can be projected from the image coordinate system to the world coordinate system, and in the world coordinate system, the actual size of each pixel corresponding to the actual space can be obtained.

A5. According to the actual size and the pixel displacement, determine the vehicle travel distance corresponding between the next frame image and the previous frame image, and obtain the first mileage traveled by the vehicle.

By mapping the pixel displacement of the target pixel point to the world coordinate system, the corresponding vehicle displacement between the next frame image and the previous frame image can be obtained, that is, the vehicle travel distance. Specifically, the product of the pixel displacement and the actual size can be calculated to obtain the vehicle displacement, that is, the first mileage.

In the above steps A1-A5, the actual size of each pixel is determined according to the conversion relationship between the pixel coordinate system and the world coordinate system, and the actual size is combined with the pixel displacement in the first image to obtain the pixel displacement corresponding to the actual space The displacement in , and then get the first mileage. This calculation method is not limited by satellite positioning signals or network signals, and the accuracy of the calculation results is not affected by the speed and state of the vehicle. The first mileage is calculated using the mapping relationship between pixels and the actual space, and there is almost no error. Higher precision.

B1. According to the inertial navigation data output by the inertial navigation device, determine the inertial navigation mileage traveled by the vehicle after the last acquisition of the first image.

Specifically, the instantaneous velocity V ₀ of the vehicle at the moment when the first image was acquired last time may be calculated, and the inertial navigation data of the vehicle may be acquired at the same time. According to the acceleration and attitude angle in the inertial navigation data, determine the acceleration component a of the vehicle in the driving direction; according to the instantaneous velocity V ₀ , the acceleration component a, and the time difference t between two adjacent acquisitions of the first image, calculate the above The mileage traveled by the vehicle after the first image is acquired once, and the inertial navigation mileage S ₁ is obtained. The formula for calculating the inertial mileage S ₁ is as follows:

S ₁ =V ₀ t+1/2*at ² (Formula 3)

B2. According to the matching pixel points between the currently acquired first image and the last acquired first image, determine the visual mileage traveled by the vehicle after the last acquired first image.

The visual mileage is the first mileage in step 102, so step B2 can refer to step 102, which will not be repeated here.

B3. Taking the average value of the inertial navigation mileage and the visual mileage as the first mileage traveled by the vehicle after the last acquisition of the first image.

Assuming that the obtained visual mileage is S ₂ , the calculation formula of the first mileage S is as follows:

S=(S ₁ +S ₂ )/2 (Formula 4)

In the above steps B1-B3, the mileage calculated according to the inertial navigation device and the mileage calculated according to the first image are averaged to obtain the first mileage, which can effectively reduce the error caused by any mileage calculation method and improve the Accuracy of the first mile.

603. Acquire inertial navigation data output by an inertial navigation device installed on the vehicle when the vehicle is detected to be slipping.

There are two reasons for vehicle skidding, one is that the road surface on which the vehicle is driving is abnormal, such as puddles, mud, etc.; the other is the acceleration and deceleration of the vehicle itself.

When the vehicle slips, the method of using the first image to calculate the first mileage will no longer be accurate. The vehicle slipping will cause the speed of the vehicle to become slower or faster, thereby affecting the matching of adjacent pixels of the first image.

For example, if the speed of the vehicle is too fast, the first image taken by the first camera will be blurred, or the number of matching pixels between two adjacent first images will be reduced or even no matching pixels will exist, which will cause adjacent The two first images do not match, so the first mileage cannot be calculated.

At this time, the inertial navigation data output by the inertial navigation device can be used to calculate the mileage traveled by the vehicle.

The inertial navigation system is a navigation parameter calculation system with gyroscope and accelerometer as sensitive devices. The system establishes a navigation coordinate system based on the output of the gyroscope, and calculates the velocity and position of the carrier in the navigation coordinate system based on the output of the accelerometer.

604. Calculate a second mileage traveled by the vehicle during the skid period of the vehicle according to the inertial navigation data.

The basic working principle of the inertial navigation device is based on Newton's laws of mechanics. By measuring the acceleration of the vehicle in the inertial reference system, integrating it with time, and transforming it into the navigation coordinate system, you can get the acceleration in the navigation coordinate system. information such as speed, yaw angle and position.

According to the acceleration in the inertial navigation data, the initial speed of the vehicle and the time interval of the vehicle skidding, the second mileage traveled by the vehicle during the skidding period can be obtained.

Optionally, the inertial navigation data includes: the acceleration of the vehicle at the current moment and the direction and attitude angle of the vehicle, and step 104 may include the following steps C1-C3:

C1. Determine the current speed of the vehicle according to the first mileage calculated last time and the time period corresponding to the first mileage.

Specifically, by dividing the first mileage calculated last time and the time period corresponding to the first mileage, an estimated value of the vehicle's speed at the current moment can be obtained, and the estimated value is used as the initial speed V ₀ .

C2. Calculate an acceleration component of the vehicle in a traveling direction according to the attitude angle and the acceleration of the vehicle.

Decompose the acceleration of the vehicle in the vertical direction and the driving direction of the vehicle, and take the acceleration component of the vehicle in the driving direction as the vehicle acceleration a.

C3. Determine a second mileage traveled by the vehicle during the vehicle skid period according to the current speed of the vehicle and the acceleration component, and a corresponding time period during the vehicle skid period.

The corresponding time period t during vehicle skidding is determined. Substituting V ₀ , a, and t into Formula 3, the second mileage traveled by the vehicle can be obtained.

605. Determine the current location of the vehicle according to the first mileage and the second mileage.

Before the current time point, there may be a period of time in which the calculation of the first mileage is successful, and a period of time in which the calculation of the first mileage fails, and there may be multiple first mileages and second mileages. The total mileage of the current vehicle can be obtained by calculating the sum of all the first mileage and the second mileage.

Acquiring the initial position of the vehicle when the first image is taken; starting from the initial position, adding the displacement of the total mileage in the forward direction of the vehicle to obtain the current position of the vehicle.

When calculating the location of the vehicle next time, you only need to add the corresponding total mileage to the previous location. The location of each moment during the driving of the vehicle can be calculated by the above method.

To sum up, in the embodiment of the present invention, a plurality of first images captured by the first camera are obtained; at least according to the currently obtained first image, the first time the vehicle travels after the last acquisition of the first image is calculated. A mileage; when detecting that the vehicle is slipping, obtain the inertial navigation data output by the inertial navigation device installed on the vehicle; calculate the first mileage of the vehicle during the slipping period of the vehicle according to the inertial navigation data Two mileages: according to the first mileage and the second mileage, determine the current location of the vehicle. In this scheme, there is no need to use satellite signals or network signals, and the first image and inertial navigation device can be used to achieve high-precision positioning in the tunnel, and the method of using inertial navigation device for positioning when the first mileage calculation fails is given. method, which improves the success rate of positioning.

FIG. 7 is a flowchart of steps of a second positioning method provided by an embodiment of the present invention.

The methods include:

701. Acquire multiple first images taken by the first camera; the first camera is installed on a moving vehicle and takes pictures continuously, and the number of matching pixels of two adjacent first images is greater than or equal to the target number .

In the embodiment of the present invention, step 701 may refer to step 601, which will not be repeated here.

702. Calculate a first mileage traveled by the vehicle after the first image was acquired last time based on at least the currently acquired first image.

In the embodiment of the present invention, step 702 may refer to step 602, which will not be repeated here.

703. The driving vehicle is also provided with a third camera, the third camera is used to photograph the mileage recognition object, and the photographing time points of the first camera and the third camera are consistent; for the third camera The content in the captured third image is identified.

Due to the unavoidable noise information in the first image, the calculation of the first mileage will generate errors, and the errors will be accumulated many times, resulting in large deviations. The encoder calculates the second mileage by accumulative counting, and the error of the second mileage is also relatively large due to problems such as wheel slipping and idling that may occur. In order to reduce the error of mileage calculation, a mechanism of fixed distance calibration is introduced. Calibrate the calculated first and second mileages by using mileage recognition objects that are commonly set up in tunnels, such as road signs, distance reminder signs, etc.

A third camera needs to be introduced to shoot at the mileage identification object in the tunnel, so as to use the mileage data in the captured third image to calibrate the calculated mileage.

Optionally, the mileage recognition object includes one of milestones on the side of the tunnel and milestones on the tunnel floor.

Milestones in the tunnel may be located on both sides of the road on the side of the tunnel, and may be upright road piles, or mileage text sprayed on the wall, or may be mileage text sprayed on the tunnel floor.

The first camera and the third camera use the same trigger signal, so that the shooting time of the two cameras can be consistent. The camera head of the third camera can be aimed at any of the above positions for continuous shooting. In the plurality of third images captured by the third camera, whether a milestone is present is identified.

704. If the mileage identification object is identified, correct the first mileage according to the mileage in the mileage identification object.

Continuously identify a plurality of third images, and if it is recognized that the current third image includes a mileage identification object, the currently calculated first mileage is corrected by using the mileage in the mileage identification object.

For example, if the currently calculated first mileage value is 99.9m, and the currently acquired third image contains a mileage recognition object with a mileage of 100m, the first mileage is corrected to 100m.

Every time the mileage recognition object is recognized in the third image, the mileage number therein can be used to correct the current first mileage, so that the first mileage can maintain the actual accuracy, thereby ensuring the accuracy of positioning.

705. Detect whether the wheel encoder installed on the vehicle outputs pulses, and detect whether the vehicle is displaced.

706. If the wheel encoder outputs pulses, determine that the vehicle is slipping when it is detected that the vehicle does not move.

707. If the wheel encoder does not output pulses, determine that the vehicle is slipping when it is detected that the vehicle is displaced.

In step 705-step 707, vehicle skidding is divided into two situations. The first kind of skidding is in-situ skidding. In this case, the wheels rotate, but the vehicle does not produce displacement; the second kind of skidding is skidding during driving. The lower wheels do not rotate, but the vehicle generates displacement. What is detected in step 706 is the first type of slipping, and what is detected in step 707 is the second type of slipping.

No matter what kind of slippage will affect the calculation of the first mileage traveled by the vehicle using the first image.

Optionally, the method for determining whether the vehicle is displaced includes the following steps D1-D3:

D1. Determine the similarity between the first image currently acquired and the first image acquired last time.

D2. If the similarity is greater than or equal to a first threshold, then determine that the vehicle has not been displaced;

D3. If the similarity is less than a first threshold, determine that the vehicle has been displaced.

In step D1-step D3, it is determined whether the vehicle is displaced according to the similarity of the first image. If the similarity between the two adjacent first images is very high, it means that the same object is captured twice by the first camera, which means that the vehicle has not moved. If the similarity between the two adjacent first images is not high, it means that the objects captured in the two shots are different, and it means that the vehicle has moved.

The first threshold can be preset according to actual conditions, for example, the first threshold can be set to 0.99.

708. Acquire inertial navigation data output by an inertial navigation device installed on the vehicle when it is detected that the vehicle is slipping.

In the embodiment of the present invention, step 708 may refer to step 603, which will not be repeated here.

709. Calculate the second mileage traveled by the vehicle during the skid period of the vehicle according to the inertial navigation data.

In the embodiment of the present invention, step 709 may refer to step 604, which will not be repeated here.

710. Correct the second mileage according to the mileage in the mileage identification object.

Similarly, the second mileage can also be corrected by using the mileage number in the mileage identification object, which is similar to the process of correcting the first mileage.

711. Determine the current location of the vehicle according to the first mileage and the second mileage.

In the embodiment of the present invention, step 711 may refer to step 605, which will not be repeated here.

712. The vehicle is further provided with a plurality of second cameras, and the second cameras are used to continuously photograph the side walls of the tunnel, and/or the top arc of the tunnel, and/or the ground of the tunnel; obtain the photographs taken by the second cameras A second image of , and identify the target object from the second image.

The second camera can be a camera array with multiple cameras facing in different directions, and is used to collect images of tunnel lining sections in all directions and in a large range. Specifically, the second camera may be used to photograph the side wall of the tunnel, and/or the top arc of the tunnel, and/or the ground of the tunnel.

The industrial computer recognizes the target object from the second image taken by the second camera, and the target object is the lesion on the surface of the tunnel.

Diseases on the tunnel surface may include lining cracks, water leakage, freezing damage, lining spalling, tunnel deformation, pavement damage, etc. By means of an image recognition algorithm, the aforementioned lesion can be identified in the second image.

713. When the target object is recognized, use the second image where the target object is located as a target image.

When a disease is identified, the second image containing the disease is used as the target image.

714. According to the current location of the vehicle, determine the location of the vehicle when the target image is captured.

According to the current location of the vehicle, the location of the vehicle at each moment can be obtained. Therefore, the location of the vehicle when shooting the target object is also easy to obtain.

715. Determine the location of the target image in the tunnel according to the location of the vehicle when the target picture is taken.

The location of the disease in the tunnel can be realized by locating the target image. By determining the location of the vehicle when the target image is captured, the location of the target image in the tunnel can be roughly determined.

To sum up, in the embodiment of the present invention, it is determined whether the vehicle is slipping according to whether the encoder sends a pulse and whether the vehicle is displaced, and the two situations of the vehicle slipping are fully considered, and the detection result is relatively accurate; and, according to two adjacent The similarity of the first image determines whether the vehicle is displaced, effectively utilizing the characteristics of the first image, and the accuracy of the judgment result is high; at the same time, according to the location of the vehicle when the target picture is taken, it is determined that the target image is in the tunnel. The positioning in the target image enables the user to quickly determine the positioning of the target image, which improves the work efficiency; in addition, the mileage in the object is recognized by the mileage, and the current calculated first mileage is corrected, so that the first mileage can be kept in line with the actual The mileage is the same, which further improves the accuracy of positioning.

FIG. 8 is a positioning device provided by an embodiment of the present invention, which is set in an electronic device, and the electronic device may be the industrial computer in FIG. 1 .

As shown in Figure 8, the device 300 includes:

The first image acquisition module 301 is configured to acquire a plurality of first images taken by the first camera; the first camera is arranged on a moving vehicle and continuously takes pictures, and the matching pixels of two adjacent first images The number of points is greater than or equal to the target amount;

The first mileage calculation module 302 is configured to calculate the first mileage traveled by the vehicle after the last acquisition of the first image based on at least the currently acquired first image;

The inertial navigation data acquisition module 306 is used to acquire the inertial navigation data output by the inertial navigation device installed on the vehicle when the vehicle is detected to be slipping;

The second mileage calculation module 304 is configured to calculate the second mileage traveled by the vehicle during the skid period of the vehicle according to the inertial navigation data;

The location module 305 is configured to determine the current location of the vehicle according to the first mileage and the second mileage.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

Fig. 5 is a structural block diagram of an electronic device 400 provided by an embodiment of the present invention. For example, the electronic device 400 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.

5, electronic device 400 may include one or more of the following components: processing component 402, memory 404, power component 406, multimedia component 408, audio component 410, input/output (I/O) interface 412, sensor component 414 , and the communication component 416 .

The processing component 402 generally controls the overall operations of the electronic device 400, such as those associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 402 may include one or more modules that facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402 .

Memory 404 is configured to store various types of data to support operations at device 400 . Examples of such data include instructions for any application or method operating on the electronic device 400, contact data, phonebook data, messages, pictures, videos, and the like. The memory 404 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

The power supply component 406 provides power to various components of the electronic device 400 . Power components 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 400 .

The multimedia component 408 includes a screen providing an output interface between the electronic device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect a duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 408 includes a front camera and/or a rear camera. When the device 400 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 410 is configured to output and/or input audio signals. For example, the audio component 410 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 400 is in operation modes, such as call mode, recording mode and voice recognition mode. Received audio signals may be further stored in memory 404 or sent via communication component 416 . In some embodiments, the audio component 410 also includes a speaker for outputting audio signals.

The I/O interface 412 provides an interface between the processing component 402 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

Sensor assembly 414 includes one or more sensors for providing various aspects of status assessment for electronic device 400 . For example, the sensor component 414 can detect the open/closed state of the device 400, the relative positioning of components, such as the display and the keypad of the electronic device 400, the sensor component 414 can also detect the electronic device 400 or a component of the electronic device 400 The position change of the electronic device 400, the presence or absence of user contact with the electronic device 400, the orientation or acceleration/deceleration of the electronic device 400 and the temperature change of the electronic device 400. The sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 414 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 414 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 416 is configured to facilitate wired or wireless communication between the electronic device 400 and other devices. The electronic device 400 can access a wireless network based on a communication standard, such as WiFi, an operator network (such as 2G, 4G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 416 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 416 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

2. Application examples. In order to prove the creativity and technical value of the technical solution of the present invention, this part is the application example of the claimed technical solution on specific products or related technologies.

The electronic equipment 400 provided by the present invention can be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGA), controller, microcontroller, microprocessor, or other electronic component implementation for performing the method described above.

The positioning method provided by the present invention can be applied to a non-transitory computer-readable storage medium including instructions, such as the memory 404 including instructions, and the above instructions can be executed by the processor 420 of the electronic device 400 to complete the above method. For example, the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

Other embodiments of the invention will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present invention, these modifications, uses or adaptations follow the general principles of the present invention and include common knowledge or conventional technical means in the technical field not disclosed in this disclosure . The specification and examples are to be considered exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

It should be understood that the present invention is not limited to the precise constructions which have been described above and shown in the accompanying drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.

Claims (10)

1. A positioning method, characterized in that the positioning method comprises: the method comprises the steps that a camera arranged on a running vehicle acquires images continuously photographed, and a first driving mileage of the vehicle is calculated according to the currently acquired images and calibration parameters of the camera;

calculating a second mileage traveled by the vehicle according to first mileage calculation failure data output by an encoder arranged on the vehicle;

and determining the current location of the vehicle according to the first mileage and the second mileage.

2. The method of claim 1, further comprising:

acquiring a plurality of first images shot by a first camera; the first cameras are arranged on a running vehicle and carry out continuous photographing, and the number of matched pixel points of two adjacent first images is larger than or equal to the target number;

according to the currently acquired first image and the calibration parameters of the first camera, calculating a first mileage of the vehicle after the first image is acquired last time;

when the first mileage calculation fails, acquiring pulse data output by an encoder arranged on the vehicle;

calculating a second mileage traveled by the vehicle during the first mileage calculation failure based on the pulse data;

and determining the current location of the vehicle according to the first mileage and the second mileage.

3. The positioning method according to claim 2, wherein a plurality of second cameras are further disposed on the vehicle, the second cameras are used for continuously shooting tunnel sidewalls, and/or tunnel top arcs, and/or tunnel ground, and after determining the current positioning of the vehicle, the method further comprises:

acquiring a second image shot by the second camera, and identifying a target object from the second image;

when the target object is identified, taking a second image where the target object is as a target image;

determining the location of the vehicle when the target image is shot according to the current location of the vehicle;

determining the positioning of the target image in the tunnel according to the positioning of the vehicle when the target image is shot;

the currently acquired first image is a next frame image, the last acquired first image is a previous frame image, and the first mileage of the vehicle after the first image is acquired last time according to the currently acquired first image and the calibration parameters of the first camera includes:

determining calibration parameters of the first camera according to the plurality of first images and matching feature points between the adjacent first images;

determining a homography matrix and a distortion parameter matrix according to the calibration parameters of the first camera; obtaining a conversion relation between a pixel coordinate system and a world coordinate system of the first image according to the homography matrix and the distortion parameter matrix;

determining pixel displacement of a target pixel point in the next frame image in an image relative to the target pixel point in the previous frame image;

determining the actual size of each pixel point in the first image corresponding to the actual space according to the conversion relation;

determining a corresponding vehicle driving distance between the next frame image and the previous frame image according to the actual size and the pixel displacement to obtain a first driving mileage of the vehicle;

the pulse data including a time stamp of an output pulse and a number of pulses, the calculating a second mileage traveled by the vehicle during the first mileage calculation failure based on the pulse data, including:

determining a total number of pulses output during the first mileage calculation failure according to the timestamp and the number of pulses;

calculating a second range traveled by the vehicle during the failure of the first range calculation based on the total number of pulses and a circumference of the wheel.

4. The positioning method according to claim 2, further comprising, before acquiring pulse data output from an encoder provided on the vehicle:

when a first abnormal condition is detected to occur, determining that the first mileage calculation is failed;

the first abnormal condition includes at least one of the following conditions: the method comprises the steps that the first image cannot be acquired, foreign matters exist in a target area on the first image, the brightness of the first image is larger than or equal to a first threshold value, and the brightness of the first image is smaller than a second threshold value; the number of matched pixel points included in two adjacent first images is smaller than the target number;

the driving vehicle is further provided with a third camera for shooting a mileage identifying object, the shooting time points of the first camera and the third camera are the same, and after the first image is obtained last time according to the currently obtained first image, the driving vehicle further comprises:

identifying content in a third image taken by the third camera;

if the mileage identification object is identified, correcting the first mileage according to the mileage number in the mileage identification object;

after calculating a second mileage traveled by the vehicle during the first mileage calculation failure based on the pulse data, the method further includes:

correcting the second mileage according to the mileage in the mileage identifying object;

after calculating a second mileage traveled by the vehicle during the first mileage calculation failure from the pulse data, further comprising:

when the second mileage calculation failure is encountered, determining a first time period length corresponding to the second mileage calculation failure period;

acquiring the mileage of the vehicle running for the last first time period;

taking the mileage traveled by the vehicle for the last first time period as a second mileage traveled by the vehicle;

after calculating a second mileage traveled by the vehicle during the first mileage calculation failure based on the pulse data, the method further includes:

when the second mileage calculation fails, acquiring inertial navigation data output by an inertial navigation device arranged on the vehicle;

and calculating a second mileage traveled by the vehicle during the second mileage calculation failure according to the inertial navigation data.

5. A positioning device, comprising:

the first image acquisition module is used for acquiring a plurality of first images shot by a first camera; the first cameras are arranged on a running vehicle and carry out continuous photographing, and the number of matched pixel points of two adjacent first images is larger than or equal to the target number;

the first mileage calculation module is used for calculating a first mileage of the vehicle after the first image is acquired last time according to the first image acquired currently and the calibration parameters of the first camera;

the pulse data acquisition module is used for acquiring pulse data output by an encoder arranged on the vehicle when the first mileage calculation fails;

the second mileage calculation module is used for calculating a second mileage driven by the vehicle during the first mileage calculation failure period according to the pulse data;

and the positioning module is used for determining the current positioning of the vehicle according to the first mileage and the second mileage.

6. The positioning method according to claim 1, further comprising:

acquiring a plurality of first images shot by a first camera; the first cameras are arranged on a running vehicle and carry out continuous photographing, and the number of matched pixel points of two adjacent first images is larger than or equal to the target number;

according to at least the first image acquired currently, calculating a first mileage traveled by the vehicle after the first image is acquired last time;

when the vehicle is detected to skid, acquiring inertial navigation data output by an inertial navigation device installed on the vehicle;

calculating a second mileage traveled by the vehicle during the vehicle slip based on the inertial navigation data;

and determining the current location of the vehicle according to the first mileage and the second mileage.

7. The positioning method according to claim 6, wherein a plurality of second cameras are further provided on the vehicle, the second cameras are used for continuously shooting tunnel side walls, and/or tunnel top arcs, and/or tunnel ground, and after determining the current positioning of the vehicle, the method further comprises:

acquiring a second image shot by the second camera, and identifying a target object from the second image;

when the target object is identified, taking a second image where the target object is as a target image;

determining the location of the vehicle when the target image is shot according to the current location of the vehicle;

determining the positioning of the target image in the tunnel according to the positioning of the vehicle when the target image is shot;

before acquiring inertial navigation data output by an inertial navigation device arranged on the vehicle, the method further comprises the following steps:

detecting whether a wheel encoder arranged on the vehicle outputs pulses or not and detecting whether the vehicle is displaced or not;

if the wheel encoder outputs pulses, determining that the vehicle skids under the condition that the vehicle is detected not to be displaced;

if the wheel encoder does not output pulses, determining that the vehicle slips under the condition that the vehicle is detected to be displaced;

the method for detecting whether the vehicle is displaced comprises the following steps:

determining the similarity between the currently acquired first image and the last acquired first image;

if the similarity is larger than or equal to a first threshold value, determining that the vehicle does not displace;

if the similarity is smaller than a first threshold value, determining that the vehicle displaces;

the inertial navigation data comprises: calculating a second distance traveled by the vehicle during the vehicle slip based on the inertial navigation data, including:

determining the speed of the vehicle at the current moment according to the first mileage obtained by the last calculation and the time period corresponding to the first mileage;

calculating an acceleration component of the vehicle in a driving direction according to the attitude angle and the acceleration of the vehicle;

determining a second mileage driven by the vehicle during the vehicle slip according to the speed and the acceleration component of the vehicle at the current moment and a corresponding time period of the vehicle slip;

the currently acquired first image is a next frame image, the last acquired first image is a last frame image, and the first mileage of the vehicle after the first image is acquired last time at least according to the currently acquired first image comprises:

determining calibration parameters of the first camera according to the plurality of first images and the matching feature points between the adjacent first images;

determining a homography matrix and a distortion parameter matrix according to the calibration parameters of the first camera; obtaining a conversion relation between a pixel coordinate system and a world coordinate system of the first image according to the homography matrix and the distortion parameter matrix;

determining pixel displacement of a target pixel point in the next frame image in an image relative to the target pixel point in the previous frame image;

determining the actual size of each pixel point in the first image corresponding to the actual space according to the conversion relation;

determining a corresponding vehicle driving distance between the next frame image and the previous frame image according to the actual size and the pixel displacement to obtain a first mileage of the vehicle;

the first mileage driven by the vehicle after the first image is last acquired at least according to the first image currently acquired comprises:

determining the driving inertial navigation mileage of the vehicle after the first image is obtained last time according to the inertial navigation data output by the inertial navigation device;

determining the visual mileage of the vehicle after the first image is obtained last time according to the matching pixel points between the first image obtained currently and the first image obtained last time;

taking the average value of the inertial navigation mileage and the visual mileage as a first mileage traveled by the vehicle after the first image was obtained last time;

the driving vehicle is further provided with a third camera for shooting a mileage identifying object, the shooting time points of the first camera and the third camera are the same, and after the first image is obtained last time according to the currently obtained first image, the driving vehicle further comprises:

identifying content in a third image taken by the third camera;

if the mileage identification object is identified, correcting the first mileage according to the mileage number in the mileage identification object;

after calculating a second mileage traveled by the vehicle during the vehicle slip based on the inertial navigation data, further comprising:

and correcting the second mileage according to the mileage in the mileage identifying object.

8. A positioning device, comprising:

the first image acquisition module is used for acquiring a plurality of first images shot by a first camera; the first cameras are arranged on a running vehicle and carry out continuous photographing, and the number of matched pixel points of two adjacent first images is larger than or equal to the target number;

the first mileage calculation module is used for calculating a first mileage traveled by the vehicle after the first image is acquired last time at least according to the currently acquired first image;

the inertial navigation data acquisition module is used for acquiring inertial navigation data output by an inertial navigation device installed on the vehicle when the vehicle is detected to skid;

the second mileage calculation module is used for calculating a second mileage of the vehicle during the vehicle skidding according to the inertial navigation data;

and the positioning module is used for determining the current positioning of the vehicle according to the first mileage and the second mileage.

9. An electronic device comprising a processor and a memory, the processor and memory coupled, the memory for storing computer program instructions that, when executed by the processor, cause the electronic device to perform the positioning method of any of claims 1 to 4 or claims 6 to 8.

10. A computer storage medium, characterized in that it has stored therein program instructions which, when run on an electronic device, cause the electronic device to perform the positioning method according to any one of claims 1 to 4 or claims 6 to 8.

Images