WO2022088103A1 - Procédé et appareil d'étalonnage d'image - Google Patents

Procédé et appareil d'étalonnage d'image Download PDF

Info

Publication number
WO2022088103A1
WO2022088103A1 PCT/CN2020/125535 CN2020125535W WO2022088103A1 WO 2022088103 A1 WO2022088103 A1 WO 2022088103A1 CN 2020125535 W CN2020125535 W CN 2020125535W WO 2022088103 A1 WO2022088103 A1 WO 2022088103A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
coordinate system
virtual image
vehicle body
coordinates
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/125535
Other languages
English (en)
Chinese (zh)
Inventor
张宇腾
周鹏程
于海
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Huawei Technologies Co Ltd
Original Assignee
Huawei Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Huawei Technologies Co Ltd filed Critical Huawei Technologies Co Ltd
Priority to PCT/CN2020/125535 priority Critical patent/WO2022088103A1/fr
Priority to CN202080004865.9A priority patent/CN112655024B/zh
Publication of WO2022088103A1 publication Critical patent/WO2022088103A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals

Definitions

  • the present application relates to the technical field of image processing, and in particular, to an image calibration method and device.
  • HUDs head up displays
  • the head-up display can project important driving information such as speed and navigation on the windshield in front of the driver, forming an image in front of the glass, so that the driver can see the speed and navigation without looking down or turning his head. driving information to improve driving safety.
  • an augmented reality head up display which combines the HUD virtual image with the real road information, thereby enhancing the driver's perception of the actual driving environment.
  • the imaging principle of AR HUD light is emitted through the light source, refracted, and finally projected onto the windshield to form a virtual HUD image.
  • the augmented reality head-up display needs to realize the human eye-HUD virtual image-road three-point line (see Figure 1a). Therefore, the position of the HUD virtual image needs to be precisely controlled according to the position of the road surface and the human eye. In order to precisely control the position of the HUD virtual image, it is necessary to accurately obtain the actual position information of the HUD virtual image.
  • Fig. 1b it is the method of measuring the position of the HUD virtual image in the prior art.
  • zoom measurement is adopted, and the HUD virtual image formed in the camera is observed by changing the focal length of the camera lens, and the focus position with the clearest imaging is the position where the HUD virtual image is located.
  • the image of the HUD virtual image on the focal plane 3 is the clearest.
  • the focal plane 3 is where the HUD virtual image is located, and the distance from the focal plane 3 to the camera is the virtual image distance.
  • the captured HUD virtual image is clear. Therefore, determining the position of the HUD virtual image according to the clear focal plane of the image will result in a large position error of the determined HUD virtual image.
  • the present application provides an image calibration method and device, which are used to improve the accuracy of the determined imaging parameters of the HUD virtual image as much as possible.
  • the present application provides an image calibration method, the method includes acquiring a first image of a target and a third image of a HUD virtual image displayed by a HUD, and acquiring a second image of the target and a fourth image of the HUD virtual image;
  • One image is obtained by shooting the target with the shooting device in the first position
  • the third image is obtained by shooting the HUD virtual image with the shooting device in the first position
  • the second image is obtained by shooting the target with the shooting device in the second position
  • the fourth image is obtained by shooting the target with the shooting device in the second position.
  • the image is obtained from the HUD virtual image taken by the camera in the second position; the first extrinsic parameter matrix of the camera in the first position can be determined according to the first image and the coordinates of the target in the vehicle body coordinate system; The coordinates of the image and the target in the vehicle body coordinate system determine the second extrinsic parameter matrix of the shooting device in the second position; it can be determined according to the first extrinsic parameter matrix, the third image, the second extrinsic parameter matrix and the fourth image.
  • the coordinates of the HUD virtual image in the vehicle body coordinate system; further, the imaging parameters of the HUD virtual image can be determined according to the coordinates of the HUD virtual image in the vehicle body coordinate system.
  • the coordinates of the HUD virtual image in the vehicle body coordinate system can be determined, and according to the coordinates of the HUD virtual image in the vehicle body coordinate system, the HUD can be further determined. Imaging parameters of the virtual image.
  • the image calibration method can simply, quickly and accurately determine the imaging parameters of the HUD virtual image.
  • the first position is different from the second position.
  • the vehicle body coordinate system may take the front wheel of the vehicle as the origin, and the forward direction or the backward direction of the vehicle as the X axis.
  • the first extrinsic parameter matrix and the second extrinsic parameter matrix may be determined in the following manner: determining the first pixel coordinates of each target point on the first image, according to the first pixel coordinates of each target point , the coordinates of each target point in the vehicle body coordinate system and the third coordinate conversion relationship to determine the first external parameter matrix; the third coordinate conversion relationship is the first pixel coordinates of each target point and each target point in the vehicle body coordinate system.
  • the second pixel coordinates of each target point on the second image according to the second pixel coordinates of each target point, the coordinates of each target point in the vehicle body coordinate system and the fourth coordinate conversion relationship , determine the second external parameter matrix, and the fourth coordinate conversion relationship is the relationship between the second pixel coordinates of each target point and the coordinates of each target point in the vehicle body coordinate system.
  • the first pixel coordinates of each target point are the coordinates of each target point on the first image in the image coordinate system.
  • the second pixel coordinates of each target point are the coordinates of each target point in the image coordinate system on the second image.
  • the HUD virtual image includes n reference points, where n is an integer greater than 1; the third pixel coordinates of the n reference points on the third image and the n reference points on the fourth image can be determined respectively.
  • the third pixel coordinates of the n reference points are the coordinates of the n reference points on the third image in the image coordinate system.
  • the fourth pixel coordinates of the n reference points are the coordinates of the n reference points on the fourth image in the image coordinate system.
  • the image coordinate system may take the upper left corner or the lower left corner of the image as the origin.
  • the imaging parameters of the HUD virtual image include, but are not limited to, any one or more of: virtual image distance (VID), horizontal field of view, vertical field of view, center position, distortion rate or rotational deformation .
  • VIP virtual image distance
  • the coordinates of each of the n reference points in the vehicle body coordinate system can be determined accurately and quickly. Based on the coordinates of the n reference points in the vehicle body coordinate system, the The imaging parameters of the HUD virtual image are determined, thereby helping to improve the accuracy and efficiency of the HUD virtual image calibration.
  • the imaging parameters for determining the HUD virtual image are introduced as follows.
  • Imaging parameter one virtual image distance.
  • the average value of the x-coordinates of at least two reference points in the vehicle body coordinate system among the n reference points on the HUD virtual image may be determined, and the x-coordinate is the forward or reverse direction of the vehicle;
  • the x coordinate of the center of the box in the vehicle body coordinate system; the absolute value of the difference between the average value and the x coordinate of the center position of the eye box in the vehicle body coordinate system is determined as the virtual image distance.
  • the second imaging parameter is the field of view.
  • the viewing angle includes a horizontal viewing angle and a vertical viewing angle.
  • the length in the horizontal direction of the HUD virtual image can be determined according to the coordinates of at least two reference points located in the same horizontal direction among the n reference points in the vehicle body coordinate system; Length and virtual image distance, determine the horizontal field of view.
  • the length of the HUD virtual image in the vertical direction may be determined according to the coordinates of at least two reference points located in the same vertical direction among the n reference points in the vehicle body coordinate system; according to the vertical length of the HUD virtual image The length in the direction and the virtual image distance determine the vertical field of view.
  • the coordinates of the center reference point among the n reference points on the HUD virtual image in the vehicle body coordinate system may be determined as the center position of the HUD virtual image.
  • the fourth imaging parameter is the distortion rate.
  • the distortion rate of the first reference point may be determined, and the first reference point is at least one of n reference points on the HUD virtual image; according to the distortion rate of the first reference point, the distortion rate of the HUD virtual image is determined Distortion rate.
  • the actual distance between the first reference point and the central reference point can be determined; then according to the central reference point and at least 4 reference points around the central reference point, the predicted distance of the first reference point is determined; Therefore, the distortion rate of the first reference point can be determined according to the actual distance and the predicted distance.
  • Imaging parameter five rotational deformation.
  • the z coordinate of the second reference point in the vehicle body coordinate system, the y coordinate of the second reference point in the vehicle body coordinate system, and the z coordinate of the third reference point in the vehicle body coordinate system are determined coordinates and the y-coordinate of the third reference point in the vehicle body coordinate system; wherein, the second reference point and the third reference point are two reference points in the same horizontal direction among the n reference points;
  • the z coordinate in the body coordinate system, the y coordinate of the second reference point in the vehicle body coordinate system, the z coordinate of the third reference point in the vehicle body coordinate system, and the y coordinate of the third reference point in the vehicle body coordinate system Determine the rotation deformation.
  • the present application provides an image calibration device, which can be used to implement the above first aspect or any one of the methods in the first aspect, and includes corresponding functional modules, which are respectively used to implement the steps in the above method.
  • the functions can be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the image calibration device may include a transceiver module and a processing module: wherein the transceiver module is used to acquire a first image of the target and a third image of the HUD virtual image displayed by the head-up display HUD, the first image The third image is obtained by photographing the target by the photographing device in the first position, and the third image is obtained by photographing the HUD virtual image by the photographing device in the first position; and the second image of the target and the fourth image of the HUD virtual image are obtained.
  • the fourth image is obtained by photographing the target by the photographing device at the second position, and the fourth image is obtained by photographing the HUD virtual image by the photographing device at the second position; the processing module is used to determine the location in the vehicle body coordinate system according to the coordinates of the first image and the target in the vehicle body coordinate system.
  • the first extrinsic parameter matrix of the photographing device at the first position; the second extrinsic parameter matrix of the photographing device at the second position is determined according to the second image and the coordinates of the target in the vehicle body coordinate system; and according to the first extrinsic parameter matrix , the third image, the second external parameter matrix and the fourth image to determine the coordinates of the HUD virtual image in the vehicle body coordinate system; and determine the imaging parameters of the HUD virtual image according to the coordinates of the HUD virtual image in the vehicle body coordinate system.
  • the HUD virtual image includes n reference points, where n is an integer greater than 1; the processing module is specifically configured to: respectively determine n reference points on the third image at the third pixel coordinates, and n The fourth pixel coordinate of the reference point on the fourth image; the first coordinate conversion relationship is determined according to the third pixel coordinates of the n reference points and the first external parameter matrix; wherein, the first coordinate conversion relationship is the The relationship between the third pixel coordinates and the coordinates of the n reference points in the vehicle body coordinate system; the second coordinate conversion relationship is determined according to the fourth pixel coordinates of the n reference points and the second external parameter matrix; The coordinate conversion relationship is the relationship between the fourth pixel coordinates of the n reference points and the coordinates of the n reference points in the vehicle body coordinate system; according to the first coordinate conversion relationship and the second coordinate conversion relationship, determine n on the HUD virtual image The coordinates of a reference point in the body coordinate system.
  • the imaging parameters of the HUD virtual image include but are not limited to any one or more of virtual image distance (VID), horizontal field of view, vertical field of view, center position, distortion rate or rotational deformation .
  • VIP virtual image distance
  • the imaging parameter includes a virtual image distance
  • the processing module is specifically configured to: determine the average value of the x-coordinates of at least two reference points in the vehicle body coordinate system among the n reference points on the HUD virtual image, x The coordinate is the forward or backward direction of the car; determine the x coordinate of the center of the eye box in the vehicle body coordinate system; the absolute value of the difference between the average value and the x coordinate of the center position of the eye box in the vehicle body coordinate system is determined as Virtual image distance.
  • the imaging parameter further includes a horizontal field of view
  • the processing module is specifically configured to: determine, according to the coordinates in the vehicle body coordinate system of at least two reference points located in the same horizontal direction among the n reference points, The length of the HUD virtual image in the horizontal direction; according to the length of the HUD virtual image in the horizontal direction and the virtual image distance, the horizontal field of view is determined.
  • the imaging parameter further includes a vertical field of view
  • the processing module is specifically configured to: determine, according to the coordinates in the vehicle body coordinate system of at least two reference points located in the same vertical direction among the n reference points, The length of the HUD virtual image in the vertical direction; according to the length of the HUD virtual image in the vertical direction and the virtual image distance, the vertical field of view is determined.
  • the imaging parameters include a center position; the processing module is specifically used to: determine the coordinates of the center reference point among the n reference points on the HUD virtual image in the vehicle body coordinate system as the center of the HUD virtual image Location.
  • the imaging parameter includes a distortion rate
  • the processing module is specifically configured to: determine the distortion rate of a first reference point, where the first reference point is at least one of n reference points on the HUD virtual image; A distortion rate of a reference point to determine the distortion rate of the HUD virtual image.
  • n is an integer greater than 5; the processing module is specifically configured to: determine the actual distance between the first reference point and the central reference point; Four reference points are used to determine the predicted distance of the first reference point; according to the actual distance and the predicted distance, the distortion rate of the first reference point is determined.
  • the imaging parameters include rotational deformation
  • the processing module is specifically configured to: determine the z coordinate of the second reference point in the vehicle body coordinate system, the y coordinate of the second reference point in the vehicle body coordinate system, The z coordinate of the third reference point in the vehicle body coordinate system and the y coordinate of the third reference point in the vehicle body coordinate system; wherein, the second reference point and the third reference point are the n reference points in the same horizontal direction Two reference points; according to the z coordinate of the second reference point in the vehicle body coordinate system, the y coordinate of the second reference point in the vehicle body coordinate system, the z coordinate of the third reference point in the vehicle body coordinate system, and the third The y-coordinate of the reference point in the body coordinate system determines the rotational deformation.
  • the processing module is specifically configured to: determine the first pixel coordinates of each target point on the first image; The coordinate and the third coordinate conversion relationship determine the first external parameter matrix; the third coordinate conversion relationship is the relationship between the first pixel coordinates of each target point and the coordinates of each target point in the vehicle body coordinate system; the processing module specifically uses In: determine the second pixel coordinates of each target point on the second image; determine the second external parameter according to the second pixel coordinates of each target point, the coordinates of each target point in the vehicle body coordinate system and the fourth coordinate conversion relationship.
  • matrix, and the fourth coordinate conversion relationship is the relationship between the second pixel coordinates of each target point and the coordinates of each target point in the vehicle body coordinate system.
  • the present application provides an image calibration device.
  • the image calibration device is used to implement the first aspect or any one of the methods in the first aspect, and includes corresponding functional modules, which are respectively used to implement the steps in the above method.
  • the functions can be implemented by hardware, or by executing corresponding software by hardware.
  • the hardware or software includes one or more modules corresponding to the above functions.
  • the image calibration apparatus may include: a transceiver and a processor.
  • the processor may be configured to support the image calibration apparatus to perform the corresponding functions of the image calibration apparatus shown above, and the transceiver to support communication between the image calibration apparatus and other devices and the like.
  • the transceiver may be an independent receiver, an independent transmitter, a transceiver with integrated transceiver functions, or an interface circuit.
  • the image calibration device may further include a memory, which may be coupled to the processor, and stores necessary program instructions and data for the image calibration device.
  • the transceiver is used to: obtain a first image of the target and a third image of the HUD virtual image displayed by the head-up display HUD, where the first image is obtained by photographing the target by the photographing device in the first position, and the third image is obtained by the camera in the first position obtained by photographing the HUD virtual image by the photographing device; and obtaining a second image of the target and a fourth image of the HUD virtual image, the second image is obtained by the photographing device at the second position photographing the target, and the fourth image is obtained by the photographing device at the second position Obtained by shooting the HUD virtual image; the processor is used to: determine the first external parameter matrix of the shooting device at the first position according to the first image and the coordinates of the target in the vehicle body coordinate system; The coordinates in the system, determine the second extrinsic parameter matrix of the shooting device at the second position; and determine the HUD virtual image in the vehicle body coordinate system according to the first extrinsic parameter matrix, the third image, the second extrin
  • the HUD virtual image includes n reference points, where n is an integer greater than 1; the processor is specifically configured to: respectively determine the coordinates of the n reference points on the third image at the third pixel, and n The fourth pixel coordinate of the reference point on the fourth image; the first coordinate conversion relationship is determined according to the third pixel coordinates of the n reference points and the first external parameter matrix; wherein, the first coordinate conversion relationship is the The relationship between the third pixel coordinates and the coordinates of the n reference points in the vehicle body coordinate system; the second coordinate conversion relationship is determined according to the fourth pixel coordinates of the n reference points and the second external parameter matrix; The coordinate conversion relationship is the relationship between the fourth pixel coordinates of the n reference points and the coordinates of the n reference points in the vehicle body coordinate system; according to the first coordinate conversion relationship and the second coordinate conversion relationship, determine n on the HUD virtual image The coordinates of a reference point in the body coordinate system.
  • the imaging parameters of the HUD virtual image include but are not limited to any one or more of virtual image distance (VID), horizontal field of view, vertical field of view, center position, distortion rate or rotational deformation .
  • VIP virtual image distance
  • the imaging parameter includes a virtual image distance
  • the processor is specifically configured to: determine the average value of the x-coordinates of at least two reference points in the vehicle body coordinate system among the n reference points on the HUD virtual image, x The coordinate is the forward or backward direction of the car; determine the x coordinate of the center of the eye box in the vehicle body coordinate system; the absolute value of the difference between the average value and the x coordinate of the center position of the eye box in the vehicle body coordinate system is determined as Virtual image distance.
  • the imaging parameter further includes a horizontal field of view;
  • the processor is specifically configured to: determine, according to the coordinates in the vehicle body coordinate system of at least two reference points located in the same horizontal direction among the n reference points, The length of the HUD virtual image in the horizontal direction; according to the length of the HUD virtual image in the horizontal direction and the virtual image distance, the horizontal field of view is determined.
  • the imaging parameter further includes a vertical field of view;
  • the processor is specifically configured to: determine, according to coordinates in the vehicle body coordinate system of at least two reference points located in the same vertical direction among the n reference points, The length of the HUD virtual image in the vertical direction; according to the length of the HUD virtual image in the vertical direction and the virtual image distance, the vertical field of view is determined.
  • the imaging parameters include a center position; the processor is specifically configured to: determine the coordinates of the center reference point among the n reference points on the HUD virtual image in the vehicle body coordinate system as the center of the HUD virtual image Location.
  • the imaging parameter includes a distortion rate
  • the processor is specifically configured to: determine a distortion rate of a first reference point, where the first reference point is at least one of n reference points on the HUD virtual image; A distortion rate of a reference point to determine the distortion rate of the HUD virtual image.
  • n is an integer greater than 5; the processor is specifically configured to: determine the actual distance between the first reference point and the central reference point; Four reference points are used to determine the predicted distance of the first reference point; according to the actual distance and the predicted distance, the distortion rate of the first reference point is determined.
  • the imaging parameters include rotational deformation
  • the processor is specifically configured to: determine the z coordinate of the second reference point in the vehicle body coordinate system, the y coordinate of the second reference point in the vehicle body coordinate system, The z coordinate of the third reference point in the vehicle body coordinate system and the y coordinate of the third reference point in the vehicle body coordinate system; wherein, the second reference point and the third reference point are the n reference points in the same horizontal direction Two reference points; according to the z coordinate of the second reference point in the vehicle body coordinate system, the y coordinate of the second reference point in the vehicle body coordinate system, the z coordinate of the third reference point in the vehicle body coordinate system, and the third The y-coordinate of the reference point in the body coordinate system determines the rotational deformation.
  • the processor is specifically configured to: determine the first pixel coordinates of each target point on the first image; The coordinate and the third coordinate conversion relationship determine the first external parameter matrix; the third coordinate conversion relationship is the relationship between the first pixel coordinates of each target point and the coordinates of each target point in the vehicle body coordinate system; the processor specifically uses In: determine the second pixel coordinates of each target point on the second image; determine the second external parameter according to the second pixel coordinates of each target point, the coordinates of each target point in the vehicle body coordinate system and the fourth coordinate conversion relationship. matrix, and the fourth coordinate conversion relationship is the relationship between the second pixel coordinates of each target point and the coordinates of each target point in the vehicle body coordinate system.
  • the present application provides an image calibration system, which includes a vehicle, a photographing device, and an image calibration device diagnosis device.
  • the image calibration device can be used to execute any one of the above first aspect or the first aspect method, and the photographing device can be used to photograph the above first image, second image, third image and fourth image.
  • the present application provides a computer-readable storage medium, in which a computer program or instruction is stored, and when the computer program or instruction is executed by the device, the image calibration device is made to perform the above-mentioned first aspect or the first aspect.
  • a method in any possible implementation of an aspect.
  • the present application provides a computer program product, the computer program product includes a computer program or an instruction, when the computer program or instruction is executed by an image calibration device, the above-mentioned first aspect or any possible implementation of the first aspect is realized method in method.
  • 1a is a schematic diagram of a human eye, a HUD virtual image, and a three-point-one-line road surface provided by the application;
  • Fig. 1b is a schematic diagram of a method of measuring the position of a HUD virtual image in the prior art
  • FIG. 2 is a schematic diagram of the relationship between a pixel coordinate system and an image coordinate system provided by the application;
  • 3a is a schematic diagram of the architecture of a system provided by the application.
  • 3b is a schematic diagram of the architecture of another system provided by the application.
  • 3c is a schematic diagram of the architecture of another system provided by the application.
  • FIG. 3d is a schematic diagram of an application scenario provided by this application.
  • FIG. 4 is a schematic flowchart of a method of an image calibration method provided by the present application.
  • FIG. 5 is a schematic diagram of a target provided by the application.
  • Fig. 6 is a kind of HUD virtual image schematic diagram provided by this application.
  • FIG. 7 is a schematic diagram of imaging parameters of a HUD virtual image provided by the application.
  • Fig. 8 is a kind of principle schematic diagram of generating ghosting provided by this application.
  • FIG. 9 is a schematic structural diagram of an image calibration device provided by the application.
  • FIG. 10 is a schematic structural diagram of an image calibration device provided by the present application.
  • the current measurement of the position of the HUD virtual image adopts the zoom measurement method, and the clearest focus position of the HUD virtual image is determined as the position where the HUD virtual image is located, and the distance from the clearest position of the HUD virtual image to the camera is the virtual image distance.
  • the camera since the camera has a certain depth of field range, within the depth of field, the captured HUD virtual image is clear. Therefore, determining the position of the HUD virtual image according to the clear focal plane of the image will result in a large position error of the determined HUD virtual image.
  • the present application provides an image calibration method, and the image calibration method in the present application can accurately and quickly calibrate the HUD virtual image.
  • the image calibration method provided by the present application will be described in detail below with reference to the accompanying drawings.
  • the world coordinate system was introduced to describe the position of the object in the real world. It is the absolute coordinate system of the objective three-dimensional world. Because the camera is placed in three-dimensional space, the reference coordinate system of the world coordinate system is needed to describe the position of the camera, and it is used to describe the position of any other object placed in this three-dimensional coordinate.
  • Use (X w , Y w , Z w ) Represents the coordinate value of the object in the world coordinate system.
  • the camera coordinate system also known as the optical center coordinate system, is a coordinate system established on the camera. It is defined to describe objects from the camera's point of view. It is used as an intermediate link between the world coordinate system and the image coordinate system (or pixel coordinate system).
  • the unit is m. Taking the optical center of the camera lens as the coordinate origin, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system, respectively, and the optical axis of the camera is the Z-axis, and its coordinates are represented by (X c , Y c , Z c ) value.
  • the image coordinate system is a two-dimensional rectangular coordinate system on the image plane.
  • the origin of the image coordinate system is the intersection of the optical axis of the lens and the image plane (also called the principal point), and the x and y axes of the image coordinate system are parallel to the X and Y axes of the camera coordinate system, respectively. Coordinate values are represented by (x, y).
  • An image coordinate system expresses the position of a pixel in an image in physical units, such as millimeters.
  • the pixel coordinate system is a two-dimensional rectangular coordinate system commonly used in image processing, which reflects the arrangement of pixels in the camera's charge coupled device (CCD)/complementary metal oxide semiconductor (CMOS) chip. .
  • the unit is pieces (number of pixels).
  • the upper left or lower left corner of the image plane is used as the origin, and the u-axis and v-axis are parallel to the X-axis and Y-axis of the image coordinate system, respectively.
  • Column, the ordinate v represents the row where the pixel is located.
  • the images collected by the camera are firstly in the form of standard electrical signals, and then converted into digital images through analog-to-digital conversion.
  • the storage form of each image is an array of P ⁇ Q, and the value of each element in the image of P row and Q column represents the grayscale of the image point.
  • Each such element is called a pixel, and the pixel coordinate system is the image coordinate system in pixels.
  • FIG. 2 is a schematic diagram of the relationship between a pixel coordinate system and an image coordinate system provided by the present application.
  • the pixel coordinate system has a translational relationship with the image coordinate system.
  • (u 0 , v 0 ) are the coordinates of the origin (principal point) of the image coordinate system
  • dx and dy are the physical dimensions of each pixel on the x-axis and y-axis, respectively.
  • the internal parameter matrix N can be understood as each value in the matrix is only related to the internal parameters of the camera, and does not change with the position of the object.
  • the transformation from three-dimensional coordinates to two-dimensional coordinates that is, the projection perspective process (using the central projection method to project the object onto the projection surface, so as to obtain a single-sided projection map that is closer to the visual effect, that is, the scene seen by the human eye is close to An imaging method of big, far and small).
  • f represents the focal length when the camera captures an image, that is, the distance between the image plane and the origin of the camera coordinate system
  • Z c represents the distance relationship between the photographer and the shooting device, which is a known number.
  • the world coordinate system and the camera coordinate system do not coincide.
  • the coordinates of the point must first be converted to the camera coordinate system. Any two three-dimensional coordinates can be converted by rotation and translation, and the process of converting a rigid body from the world coordinate system to the camera coordinate system can also be obtained by rotation and translation.
  • the coordinate of point P in the world coordinate system be X w
  • the vertical distance from P to the optical center is s
  • the coordinate on the image plane is x
  • the relative rotation between the world coordinate system and the camera coordinate system is the matrix R (R is a rotation matrix with three rows and three columns)
  • the relative displacement is a vector T (three rows and one column).
  • the homogeneous coordinate matrix composed of a rotation matrix and a translation vector is expressed as follows:
  • (X w , Y w , Z w , 1) are the homogeneous coordinates of the world coordinate system
  • (X c , Y c , Z c , 1) are the homogeneous coordinates of the camera coordinate system. It should be understood that since the transformation matrix between the world coordinate system and the camera coordinate system has nothing to do with the camera, it is also called the external parameter matrix.
  • camera calibration In the process of image measurement and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a space object and its corresponding point in the image, it is necessary to establish a geometric model of camera imaging, and these geometric model parameters are camera parameters.
  • the purpose of camera calibration is to obtain the internal parameters of the camera (such as the internal parameter matrix) and external parameters (such as the outer gap), and the process of solving the parameters can be called camera calibration.
  • the eye box usually refers to the area where the driver's eyes can see the entire displayed image.
  • the general eye box size is 130mmx50mm. Due to the height of different drivers, the eye box needs to have a moving range of about ⁇ 50mm in the vertical direction.
  • the human eye can see the area of the clear HUD virtual image in the scope of the eye box. Referring to Figure 1a above, if the human eye is aligned with the center of the eye box, a complete and clear HUD virtual image can be obtained. As the eye moves left and right or up and down, at some point in each direction, the image will deteriorate until it becomes unacceptable, i.e. beyond the eye box. Image distortion, wrong color rendering, or even no display may occur in areas beyond the eye box.
  • FIG. 3a is a schematic diagram of the architecture of an applicable system of the present application.
  • the system may include a target, a vehicle, a photographing device and a fixed component, wherein the vehicle includes an AR-HUD, an AR-HUD -
  • the vehicle includes an AR-HUD, an AR-HUD -
  • the specific structure of the HUD can be seen in Figure 3b or Figure 3c.
  • the HUD virtual image generated by AR-HUD can be projected in the driver's front view.
  • the main principle of AR-HUD is to use multiple curved mirrors or plane emitters to amplify the HUD virtual image generated by the picture generation unit (PGU) and reflect it to a certain position outside the car, that is, to reflect the driver's In the forward field of view (eyebox range), the driver is presented with an image that is a certain distance (eg 2 to 20m) away from the road.
  • the actual position of the HUD virtual image is determined by the HUD's optical system. In theory, the navigation lane lines projected by the AR HUD and related warning information should fit the actual road as closely as possible, preferably without errors.
  • the AR HUD has an imaging distance of more than 7.5 meters according to the actual driving needs of the vehicle, so that the virtual image of the HUD can be superimposed with the object or the real road scene to form an augmented reality effect, so that the driver can
  • the prompt information is obtained while observing the real environment, and there is no longer a blind spot for vision. It should be understood that if the AR-HUD only displays some vehicle speed and prompt information, it does not need to care too much about the position of the HUD virtual image, but if it involves navigation, advanced driving assistant system (ADAS) information, etc., you need to Get the exact position of the HUD virtual image.
  • ADAS advanced driving assistant system
  • the photographing device is arranged in the eye box area, wherein the eye box range is usually about 10 cm.
  • the photographing device may be, for example, a camera, and the system may include one photographing device or may include two photographing devices, wherein the two photographing devices are located at different positions and are both arranged in the eye box area.
  • the fixing assembly is used to fix the camera in a certain position.
  • the fixing device can be, for example, a robotic arm, or a slide rail.
  • a camera can be fixed at different positions through a robotic arm or a slide rail, see Figure 3b; alternatively, it can also include two fixing components, one fixing component can fix the photographing device in one position, and two fixing components Two cameras can be fixed in two positions, see Figure 3c.
  • the target may include at least 6 target points, and the coordinates of the target points in the vehicle body coordinate system are known, and the HUD virtual image can be calibrated by the target.
  • the distance between the target and the vehicle can be determined according to the specific scene.
  • the shape of the target may be circular, or square, or other regular or irregular shapes, and the circular targets in Figures 3a to 3c are for illustration only.
  • Fig. 3a, Fig. 3b and Fig. 3c can be applied to the scene of HUD virtual image calibration on the vehicle AR-HUD production line, please refer to Fig. 3d.
  • FIG. 3d is a scene to which this application can be applied.
  • test equipment and vehicles can be included.
  • the test equipment can be connected to the vehicle through the on-board diagnostic system (OBD) port on the vehicle.
  • OBD on-board diagnostic system
  • the test equipment is plugged into the OBD port of the vehicle, enabling communication between the test equipment and the vehicle.
  • the OBD is usually installed in the vehicle and can be used to record the performance information of the vehicle in real time, wherein the interface through which the OBD communicates with the test equipment can be called an OBD port.
  • Test equipment is a professional instrument or system dedicated to vehicle inspection, that is, the test equipment can be used to obtain information about the vehicle. For example, it can be used to detect the performance of the vehicle, and the performance information of the vehicle (such as the imaging parameters of the HUD virtual image) can be obtained.
  • the test equipment can realize the test of the vehicle through the developed test software. It can also be understood that the equipment with the test software installed can be understood as the test equipment. For example, a personal computer (PC) installed with testing software, or a tablet computer, or a special device, etc., wherein the special device such as a diagnostics tester (DT), DT can also be called a tester or a vehicle diagnostics or upper instrument. Further, the test equipment can also present various test information in the form of a graphical interface.
  • PC personal computer
  • DT diagnostics tester
  • FIG. 4 exemplarily shows an image calibration method provided by the present application.
  • the test equipment in this method may be the test equipment in the above-mentioned FIG. 3d, and the AR-HUD may be the AR-HUD in any of the above-mentioned embodiments of FIG. 3a to FIG. 3d.
  • the method includes the following steps:
  • Step 401 the testing device obtains the coordinates (x T , y T , z T ) of the target point on the target in the vehicle body coordinate system.
  • This step 401 is an optional step.
  • the coordinates of the target point on the target in the vehicle body coordinate system are (x T , y T , z T ).
  • the target includes at least 6 target points, and at least 3 target points are not collinear. That is, the target is a target that has been calibrated.
  • T can be 1 to 6, please refer to FIG. 5 , which is a schematic diagram of a target provided in the present application.
  • the target includes 6 reference points.
  • the coordinates of the 6 target points in the vehicle body coordinate system are: (x 1 , y 1 , z 1 ), (x 2 , y 2 , z 2 ), (x 3 , y 3 , z 3 ), (x 4 , y 4 , z 4 ), (x 5 , y 5 , z 5 ), (x 6 , y 6 , z 6 ).
  • Step 402 the test device sends a lighting instruction to the AR-HUD.
  • the AR-HUD receives the lighting instruction from the test equipment, and lights up according to the lighting instruction, so that the AR-HUD displays the HUD virtual image.
  • This step 402 is an optional step.
  • the HUD virtual image can be displayed at a certain position in front of the driver.
  • the HUD virtual image may be referred to as a calibration image or a test image.
  • C 0 is the center point of the HUD virtual image
  • C 1 , C 2 , C 3 , and C 4 are four points on the top, bottom, left, right, and right of C 0 , and the distances from C 0 are all equal.
  • the distances from C 0 may also be unequal, and the specific distance may be determined according to the size of the HUD virtual image. For example, it can be 1/4 of the length and width of the HUD virtual image.
  • E 1 , E 2 , E 5 , and E 6 are the four vertices of the edge of the HUD virtual image, and E 3 and E 4 are the center points of the vertical sides.
  • C 0 , C 1 , C 2 , C 3 , and C 4 may represent central regions
  • E 1 , E 2 , E 3 , E 4 , E 5 , and E 6 may represent edge regions.
  • the test equipment can send instructions to the HUD through the OBD port, and the HUD lights up according to the instructions to display the HUD virtual image.
  • the size of the HUD virtual image is related to the type of AR-HUD, so the number of reference points on the HUD virtual image of different AR-HUDs will also vary.
  • the number of reference points on the HUD virtual image can be set according to the requirements given at the factory. For example, if the spacing between reference points is required to be less than 0.5 degrees and evenly distributed, the minimum number of reference points required on the HUD virtual image can be determined. quantity.
  • Step 401 may be performed first and then step 402 may be performed, or step 402 may be performed first and then step 403 may be performed.
  • Step 403 the testing device acquires the first image of the target and the third image of the virtual HUD image displayed by the HUD.
  • the first image is obtained by photographing the target by the photographing device in the first position
  • the third image is obtained by photographing the HUD virtual image by the photographing device at the first position.
  • the photographing device may transmit the photographed first image and the third image to the test equipment (eg, through a network).
  • Step 404 the testing device acquires the second image of the target and the fourth image of the HUD virtual image.
  • the second image is obtained by photographing the target by the photographing device in the second position
  • the fourth image is obtained by photographing the HUD virtual image by the photographing device at the second position.
  • the above-mentioned system includes two shooting devices, that is, the target and the HUD virtual image are shot by the two shooting devices, wherein one shooting device can be set at the first position, and the other is shooting
  • the device can be set in the second position, the shooting device in the first position captures the target to obtain the first image, and the HUD virtual image captures the third image; the capture device in the second position captures the target to obtain the second image, and captures the HUD virtual image to obtain the fourth image.
  • the photographing device at the first position and the photographing device at the second position may be photographed at the same time, that is, the first image and the second image may be photographed at the same time, and the third image and the fourth image may also be photographed at the same time.
  • the two photographing devices may not be photographed at the same time, which is not limited in this application.
  • the first image and the third image may be obtained by the photographing device at the first position
  • the second image and the fourth image are obtained by the photographing device at the first position.
  • the photographing device may be moved to the first position or the second position by, for example, a robotic arm or a slide rail, and may be moved left and right, or may be moved back and forth, or may be moved up and down.
  • the photographing device can transmit the second image and the fourth image obtained by photographing to the test equipment (eg, through a network).
  • both the first position and the second position are within the eye box area.
  • the driver's eyes can see a clear HUD virtual image in the eye box area. If the area of the eye box is exceeded, the driver cannot see the relevant image or see that the distortion of the image is serious.
  • the distance between the first location and the second location is less than 10 cm.
  • Step 405 the testing device determines the first pixel coordinates of each target point included in the first image
  • the first pixel coordinates of each target point are the coordinates of each target point on the first image in the image coordinate system.
  • a checkerboard is used to represent each target on the first image, and an image processing algorithm is used to identify the straight line features and black and white features of the target region, and detect parallel lines, which are determined by intersection points. target. Since the positional relationship between the target points is known (that is, the distance between the target points is known), the relationship between each target point and the intersection point in the first image can be inferred, and then each target point included in each first image can be determined. The first pixel coordinate of .
  • Step 406 the testing equipment is based on the coordinates (x T , y T , z T ) of each target point in the vehicle body coordinate system, the first pixel coordinates of each target point on the first image and a third coordinate conversion relationship to determine the first extrinsic parameter matrix of the photographing device at the first position.
  • the third coordinate conversion relationship is the relationship between the first pixel coordinates of each target point and the coordinates of each target point in the vehicle body coordinate system.
  • the pixel coordinates (u, v), the camera's internal parameter matrix, and the coordinates (X c , Y c , Z c ) in the camera coordinate system satisfy the first relationship, that is, formula 1; the coordinates in the camera coordinate system ( The second relationship is satisfied between X c , Y c , Z c ), the external parameter matrix of the camera, and the coordinates (x T , y T , z T ) in the vehicle body coordinate system, that is, formula 2.
  • (dx, dy) represents the pixel size in the pixel array of the photographing device
  • (u 0 , v 0 ) represents the center coordinates of the pixel array of the photographing device
  • f is the focal length when the photographing device captures an image
  • Z C represents the distance relationship between the photographer and the photographing device, which is a known number.
  • the first pixel coordinates of each target point There is a third coordinate conversion relationship between the coordinates (x T , y T , z T ) of each target point in the vehicle body coordinate system, that is, the first pixel coordinate of each target point Substitute the coordinates (x T , y T , z T ) of each target point in the vehicle body coordinate system into the above formula 1 and formula 2 to obtain formula 3 and formula 4.
  • the first external parameter can be determined.
  • the first pixel coordinates of the 6 target points are respectively The coordinates of the 6 target points in the vehicle body coordinate system are (x 1 , y 1 , z 1 ), (x 2 , y 2 , z 2 ), (x 3 , y 3 , z 3 ), (x 4 ) , y 4 , z 4 ), (x 5 , y 5 , z 5 ), (x 6 , y 6 , z 6 ); where, A target point corresponding to (x 1 , y 1 , z 1 ), A target point corresponding to (x 2 , y 2 , z 2 ), A target point corresponding to (x 3 , y 3 , z 3 ), A target point corresponding to (x 4 , y 4 , z 4 ), A target point corresponding to (x 5 , y 5 , z 5 ), A target point corresponding to (x 6 , y 6 , z 6 );
  • Step 407 the testing device determines the second pixel coordinates of each target point included on the second image
  • the second pixel coordinates of each target point are the coordinates of each target point in the image coordinate system on the second image.
  • step 407 reference may be made to the introduction of the above-mentioned step 405, and details are not repeated here.
  • Step 408 the testing equipment is based on the coordinates (x T , y T , z T ) of each target point in the vehicle body coordinate system and the second pixel coordinates of each target point on the second image A second extrinsic parameter matrix of the photographing device in the second position is determined.
  • the second pixel coordinates of each target point There is a fourth coordinate conversion relationship between the coordinates (x T , y T , z T ) of each target point in the vehicle body coordinate system, that is, the second pixel coordinate of each target point Substitute the coordinates (x T , y T , z T ) of each target point in the vehicle body coordinate system into the above formula 1 and formula 2 to obtain formula 5 and formula 6.
  • the second external parameter can be determined.
  • the second pixel coordinates of the 6 target points are respectively The corresponding vehicle body coordinates are (x 1 , y 1 , z 1 ), (x 2 , y 2 , z 2 ), (x 3 , y 3 , z 3 ), (x 4 , y 4 , z 4 ) , (x 5 , y 5 , z 5 ), (x 6 , y 6 , z 6 ); where, A target point corresponding to (x 1 , y 1 , z 1 ), A target point corresponding to (x 2 , y 2 , z 2 ), A target point corresponding to (x 3 , y 3 , z 3 ), A target point corresponding to (x 4 , y 4 , z 4 ), A target point corresponding to (x 5 , y 5 , z 5 ), A target point corresponding to (x 6 , y 6 , z 6 ).
  • the first extrinsic parameter matrix of the photographing device at the first position can be obtained and the second extrinsic parameter matrix of the camera at the second position That is to say, through the above steps 401 to 408, the camera position calibration is completed, and the camera position is calibrated according to the target point with known coordinates, which helps to improve the accuracy of the camera position calibration.
  • the extrinsic parameter matrix of the photographing device is related to the position of the photographing device, that is, the extrinsic parameter matrix of the first image and the third image taken by the photographing device in the first position is the same, and both are the first extrinsic parameter matrix.
  • parameter matrix; the extrinsic parameter matrix of the second image and the fourth image captured by the photographing device in the second position are the same, and both are the second extrinsic parameter matrix.
  • Step 409 the testing device determines the third pixel coordinates of the n reference points on the third image, and the fourth pixel coordinates of the n reference points on the fourth image.
  • the third pixel coordinates of the n reference points are the coordinates of the n reference points on the third image in the image coordinate system.
  • the fourth pixel coordinates of the n reference points are the coordinates of the n reference points on the fourth image in the image coordinate system.
  • the above-mentioned image coordinate systems for determining the first pixel coordinates, the second pixel coordinates, the third pixel coordinates, and the fourth pixel coordinates are the same.
  • the images captured by the same photographing device all take the lower left corner of the images (eg, the first image, the second image, the third image and the fourth image) as the origin, or all take the lower right corner as the origin.
  • the third pixel coordinate means, the fourth pixel coordinate is used Indicates that, for the manner of determining the third pixel coordinate and the fourth pixel coordinate, reference may be made to the introduction of the above step 405, and details are not repeated here.
  • Step 410 The testing device determines the coordinates of the HUD virtual image in the vehicle body coordinate system according to the first extrinsic parameter matrix, the third image, the second extrinsic parameter matrix, and the fourth image.
  • the first coordinate conversion relationship may be determined according to the third pixel coordinates of the n reference points and the first external parameter matrix; wherein, the first coordinate conversion relationship is the n The relationship between the third pixel coordinates of the reference point and the coordinates of the n reference points in the vehicle body coordinate system.
  • the testing device may determine the n pixels on the HUD virtual image according to the above formula 1, formula 2, the third pixel coordinates of the n reference points, and the fourth pixel coordinates of the n reference points The coordinates of the reference point in the vehicle body coordinate system.
  • the third pixel coordinates of the n reference points may be And the first external parameter matrix is substituted into the above formula 1 and formula 2, and formula 7 and formula 8 are obtained, and the third pixel coordinates of n reference points can be obtained.
  • a second coordinate conversion relationship can be determined according to the fourth pixel coordinates of the n reference points and the second external parameter matrix; wherein, the second coordinate conversion relationship is all the n reference points.
  • the fourth pixel coordinates of the n reference points may be And the second external parameter matrix is substituted into the above formula 1 and formula 2 to obtain formula 9 and formula 10, and the fourth pixel coordinates of the n reference points can be obtained The second coordinate conversion relationship with the coordinates (x T , y T , z T ) of the n reference points in the vehicle body coordinate system.
  • the coordinates of the n reference points on the HUD virtual image in the vehicle body coordinate system may be determined according to the first coordinate conversion relationship and the second coordinate conversion relationship determined above. That is to say, based on the above formula 7, formula 8, formula 9 and formula 10, the vehicle body coordinates (x H , y H , z H ) of each of the n reference points on the HUD virtual image can be determined, namely ( x 1 , y 1 , z 1 ), (x 2 , y 2 , z 2 )...(x n , y n , z n ).
  • the HUD virtual image is a surface existing in the three-dimensional space. If the HUD virtual image is perpendicular to the X axis, the x values of the n reference points on the HUD virtual image are the same. If the HUD virtual image is not perpendicular to the X axis, the x values of the n reference points on the HUD virtual image may be different.
  • Step 411 the testing device determines the imaging parameters of the HUD virtual image according to the coordinates of the HUD virtual image in the vehicle body coordinate system.
  • HUD virtual images When judging AR-HUD eligibility, it is necessary to provide a series of imaging parameters (ie detection indicators) of HUD virtual images, such as virtual image distance (VID), horizontal field of view, vertical field of view, center position, distortion rate One or more of , rotational deformation, etc.
  • VID virtual image distance
  • HDR horizontal field of view
  • vertical field of view center position
  • distortion rate One or more of , rotational deformation, etc.
  • the image calibration method is simple, fast and accurate.
  • the process of determining the imaging parameters of the HUD virtual image is exemplarily shown.
  • the reference point on the HUD virtual image is taken as an example in the above-mentioned FIG. 6 .
  • Imaging parameter one virtual image distance.
  • the virtual image distance refers to the distance from the HUD virtual image to the human eye, see Figure 7.
  • the average value of the x-coordinates of at least two reference points in the vehicle body coordinate system among the n reference points on the HUD virtual image may be determined, and the center of the eye box is in the vehicle body coordinate system.
  • the x coordinate in the vehicle body coordinate system; the absolute value of the difference between the average value and the x coordinate of the eyebox center position in the vehicle body coordinate system is determined as the virtual image distance. It should be noted that the coordinates (x e , y e , z e ) of the center position of the eye box in the vehicle body coordinate system have been determined during the AR-HUD design.
  • the virtual image distance can be determined by the following formula 11.
  • x e represents the x coordinate of the center position of the eye box in the vehicle body coordinate system
  • x 1 , x 2 , ... x n represents that each of the n reference points on the HUD virtual image is in the vehicle body coordinate system. the x-coordinate of .
  • the second imaging parameter is the field of view.
  • the field of view includes a horizontal field of view (H_FOV) and a vertical field of view (V_FOV).
  • H_FOV horizontal field of view
  • V_FOV vertical field of view
  • the horizontal field of view refers to the maximum visible range of the human eye in the horizontal direction
  • the vertical field of view refers to the maximum visible range of the human eye in the vertical direction.
  • the length in the horizontal direction of the HUD virtual image may be determined according to the coordinates of at least two reference points located in the same horizontal direction among the n reference points in the vehicle body coordinate system ; Determine the horizontal field of view according to the length of the HUD virtual image in the horizontal direction and the virtual image distance.
  • the horizontal field of view can be determined by the following formula 12.
  • H_FOV 2 ⁇ Arctan[half of the length of the HUD virtual image in the horizontal direction/virtual image distance]
  • the length in the vertical direction of the HUD virtual image may be determined according to the coordinates of at least two reference points located in the same vertical direction among the n reference points in the vehicle body coordinate system ; According to the length in the vertical direction of the HUD virtual image and the virtual image distance, determine the vertical angle of view.
  • the vertical viewing angle can be determined by the following formula 13.
  • V_FOV 2 ⁇ Arctan[half of the length of the HUD virtual image in the vertical direction/virtual image distance]
  • Imaging parameter three the center position of the HUD virtual image.
  • the coordinates of the center reference point among the n reference points on the HUD virtual image in the vehicle body coordinate system may be determined as the center position of the HUD virtual image.
  • the vehicle body coordinate of the reference point C 0 is the center coordinate of the HUD virtual image.
  • the fourth imaging parameter is the distortion rate of the first reference point.
  • the first reference point may be any one or any number of n reference points.
  • the reference point in the central region of the HUD virtual image is less prone to distortion.
  • C 0 , C 1 , C 2 , C 3 , and C 4 are considered to be not distorted, and the predicted distance between the central reference point and the first reference point can be determined based on the reference point that is not prone to distortion, and then The actual distance between the central reference point and the first reference point is determined, and then the distortion rate of the first reference point is determined according to the actual distance and the predicted distance.
  • the actual distance between the first reference point and the center reference point is determined, and the first reference point is determined according to the center reference point and at least four reference points around the center reference point.
  • the predicted distance of the point; the distortion rate of the first reference point can be determined by the following formula 14.
  • the center reference point C 0 and the first reference point E are determined based on reference points C 0 , C 1 , C 2 , C 3 , and C 4 that are not prone to distortion Predicted distance C 0 E 1 between 1 .
  • Distortion rate of E 1 [(actual distance C 0 E 1 /predicted distance C 0 E 1 ) ⁇ 1] ⁇ 100%.
  • the direction of the distortion can be identified by a positive or negative sign, that is, a positive sign indicates that the distortion causes the original image to be enlarged, and a negative sign indicates that the distortion causes the original image to be reduced.
  • the distortion rate of the HUD virtual image may be determined according to the distortion rate of the first reference point.
  • the distortion rate of the HUD virtual image may be determined by weighted averaging the distortion rates of the n reference points; or the distortion rate of the maximum distortion point among the n reference points may be determined as the distortion rate of the HUD virtual image;
  • Reference points in the middle of the reference points eg, the average of any one or more of reference points C 0 , C 1 , C 2 , C 3 , C 4 in FIG. 6
  • edge vertices eg, FIG.
  • the distortion ratios of any one or more of E 1 , E 2 , E 3 , E 4 , E 5 , and E 6 in 6) are used as the distortion ratio of the HUD virtual image. It should be understood that the distortion in the middle area of the HUD virtual image is relatively small, and the distortion in the edge area is relatively large.
  • the fifth imaging parameter is the degree of rotational deformation.
  • the HUD virtual image may have a degree of rotational deformation, which is represented by ⁇ .
  • ⁇ degree of rotational deformation
  • the HUD virtual image mainly rotates around the X-axis, so the rotational deformation degree ⁇ can be determined by formula 15.
  • the second reference point and the third reference point are two reference points in the same horizontal direction among the n reference points; the fourth reference point and the fifth reference point are the n reference points Two reference points in the same vertical direction among the reference points.
  • E 1 E 2 are two reference points in the same horizontal direction
  • E 3 E 4 are two reference points in the same horizontal direction
  • E 5 E 6 are two reference points in the same horizontal direction
  • the rotational deformation degrees of E 1 E 2 , E 3 E 4 , and E 5 E 6 are the same.
  • the rotational deformation degrees of the HUD virtual image are described by taking E 1 E 2 as an example.
  • z E1 represents the z coordinate of the reference point E 1 in the vehicle body coordinate system
  • z E2 represents the z coordinate of the reference point E 2 in the vehicle body coordinate system
  • y E1 represents the reference point E 1 in the vehicle body coordinate system
  • Imaging parameter six ghosting
  • the ghost ⁇ can be determined by formula 17.
  • VID represents the virtual image distance
  • E i E j represents the distance between the reference point E i and the reference point E j in the vehicle body coordinate system, which can be determined by Referring to FIG. 7 , both i and j can take any integer from 1 to 6.
  • E 1 E 2 means that i takes 1 and j takes 2.
  • C i C j represents the distance between the reference point C i and the reference point C j in the vehicle body coordinate system, which can be obtained by Referring to FIG. 7 , both i and j can take any integer from 0 to 4.
  • C 1 C 2 means i takes 1 and j takes 2.
  • the testing equipment determines the imaging parameters of the HUD virtual image, it can transmit the parameters to the HUD through the OBD port to complete the calibration of the HUD virtual image.
  • the image calibration apparatus or the test equipment includes corresponding hardware structures and/or software modules for executing each function.
  • the modules and method steps of each example described in conjunction with the embodiments disclosed in the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is performed by hardware or computer software-driven hardware depends on the specific application scenarios and design constraints of the technical solution.
  • FIG. 9 and FIG. 10 are schematic structural diagrams of possible image calibration apparatuses provided by the present application. These image calibration apparatuses can be used to implement the functions of the testing equipment in the above method embodiments, and thus can also achieve the beneficial effects of the above method embodiments.
  • the image calibration device may be the test equipment as shown in Figure 3d, or may be a module (such as a chip) applied to the test equipment.
  • the image calibration apparatus 900 includes a processing module 901 and a transceiver module 902 .
  • the image calibration apparatus 900 is used to implement the functions of the test equipment in the method embodiment shown in FIG. 4 above.
  • the transceiver module 902 is used to acquire the first image of the target and the third image of the virtual HUD image displayed by the head-up display HUD, the first image The third image is obtained by photographing the target by the photographing device in the first position, and the third image is obtained by photographing the HUD virtual image by the photographing device in the first position; and the second image of the target and the fourth image of the HUD virtual image are obtained.
  • the fourth image is obtained by shooting the target by the shooting device in the second position, and the fourth image is obtained by shooting the HUD virtual image by the shooting device in the second position; the processing module 901 is used to determine the position in the vehicle body coordinate system according to the first image and the coordinates of the target in the vehicle body coordinate system.
  • the first extrinsic parameter matrix of the photographing device at the first position; the second extrinsic parameter matrix of the photographing device at the second position is determined according to the second image and the coordinates of the target in the vehicle body coordinate system; and according to the first extrinsic parameter matrix , the third image, the second external parameter matrix and the fourth image to determine the coordinates of the HUD virtual image in the vehicle body coordinate system; and determine the imaging parameters of the HUD virtual image according to the coordinates of the HUD virtual image in the vehicle body coordinate system.
  • processing module 901 in this embodiment of the present application may be implemented by a processor or a circuit component related to the processor, and the transceiver module 902 may be implemented by a transceiver or a circuit component related to the transceiver.
  • the present application further provides an image calibration apparatus 1000 .
  • the image calibration apparatus 1000 may include a processor 1001 and a transceiver 1002 .
  • the processor 1001 and the transceiver 1002 are coupled to each other.
  • the transceiver 1002 can be an interface circuit or an input-output interface.
  • the image calibration apparatus 1000 may further include a memory 1003 for storing instructions executed by the processor 1001 or input data required by the processor 1001 to run the instructions or data generated after the processor 1001 runs the instructions.
  • the processor 1001 is used to perform the functions of the above-mentioned processing module 901
  • the transceiver 1002 is used to perform the functions of the above-mentioned transceiver module 902 .
  • processor in the embodiments of the present application may be a central processing unit (central processing unit, CPU), and may also be other general-purpose processors, digital signal processors (digital signal processors, DSP), application-specific integrated circuits (application specific integrated circuit, ASIC), field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof.
  • CPU central processing unit
  • DSP digital signal processors
  • ASIC application specific integrated circuit
  • FPGA field programmable gate array
  • a general-purpose processor may be a microprocessor or any conventional processor.
  • the method steps in the embodiments of the present application may be implemented in a hardware manner, or may be implemented in a manner in which a processor executes software instructions.
  • Software instructions can be composed of corresponding software modules, and software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (programmable ROM) , PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically erasable programmable read-only memory (electrically EPROM, EEPROM), registers, hard disks, removable hard disks, CD-ROMs or known in the art in any other form of storage medium.
  • An exemplary storage medium is coupled to the processor, such that the processor can read information from, and write information to, the storage medium.
  • the storage medium can also be an integral part of the processor.
  • the processor and storage medium may reside in an ASIC.
  • the ASIC may be located in the image calibration device.
  • the processor and the storage medium may also exist in the image calibration device as discrete components.
  • the above-mentioned embodiments it may be implemented in whole or in part by software, hardware, firmware or any combination thereof.
  • software it can be implemented in whole or in part in the form of a computer program product.
  • the computer program product includes one or more computer programs or instructions.
  • the processes or functions described in the embodiments of the present application are executed in whole or in part.
  • the computer may be a general purpose computer, a special purpose computer, a computer network, network equipment, user equipment, or other programmable apparatus.
  • the computer program or instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instructions may be downloaded from a website, computer, A server or data center transmits by wire or wireless to another website site, computer, server or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server, data center, or the like that integrates one or more available media.
  • the usable medium can be a magnetic medium, such as a floppy disk, a hard disk, and a magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); it can also be a semiconductor medium, such as a solid state drive (solid state drive). , SSD).
  • a magnetic medium such as a floppy disk, a hard disk, and a magnetic tape
  • an optical medium such as a digital video disc (DVD)
  • DVD digital video disc
  • it can also be a semiconductor medium, such as a solid state drive (solid state drive). , SSD).
  • the word "exemplary” is used to mean serving as an example, illustration, or illustration. Any embodiment or design described in this application as “exemplary” should not be construed as preferred or advantageous over other embodiments or designs. Alternatively, it can be understood that the use of the word example is intended to present concepts in a specific manner, and not to limit the application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Instrument Panels (AREA)
  • Image Analysis (AREA)

Abstract

Procédé et appareil d'étalonnage d'image, utilisés pour résoudre le problème d'une grande erreur de distance d'image virtuelle d'une image virtuelle de HUD déterminée dans l'état de la technique. Le procédé consiste : à acquérir une première image et une deuxième image d'une cible, et une troisième image et une quatrième image d'une image virtuelle de HUD, la première image et la troisième image étant photographiées par un appareil de prise de vues à une première position, et la deuxième image et la quatrième image étant photographiées par l'appareil de prise de vues à une deuxième position ; en fonction de la première image et des coordonnées de la cible dans un système de coordonnées de carrosserie de véhicule, à déterminer une première matrice de paramètres externes de l'appareil de prise de vues ; en fonction de la deuxième image et des coordonnées de la cible dans le système de coordonnées de carrosserie de véhicule, à déterminer une deuxième matrice de paramètres externes de l'appareil de prise de vues ; et en fonction de la première matrice de paramètres externes, de la troisième image, de la deuxième matrice de paramètres externes et de la quatrième image, à déterminer les coordonnées de l'image virtuelle de HUD dans le système de coordonnées de carrosserie de véhicule, de façon à déterminer des paramètres d'imagerie de l'image virtuelle de HUD. Grâce à quatre images photographiées à une première position et à une deuxième position, les coordonnées d'une image virtuelle de HUD dans un système de coordonnées de carrosserie de véhicule peuvent être déterminées, de telle sorte que des paramètres d'imagerie de l'image virtuelle de HUD peuvent être déterminés avec précision.
PCT/CN2020/125535 2020-10-30 2020-10-30 Procédé et appareil d'étalonnage d'image Ceased WO2022088103A1 (fr)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2020/125535 WO2022088103A1 (fr) 2020-10-30 2020-10-30 Procédé et appareil d'étalonnage d'image
CN202080004865.9A CN112655024B (zh) 2020-10-30 2020-10-30 一种图像标定方法及装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/125535 WO2022088103A1 (fr) 2020-10-30 2020-10-30 Procédé et appareil d'étalonnage d'image

Publications (1)

Publication Number Publication Date
WO2022088103A1 true WO2022088103A1 (fr) 2022-05-05

Family

ID=75368404

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/125535 Ceased WO2022088103A1 (fr) 2020-10-30 2020-10-30 Procédé et appareil d'étalonnage d'image

Country Status (2)

Country Link
CN (1) CN112655024B (fr)
WO (1) WO2022088103A1 (fr)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115131529A (zh) * 2022-06-15 2022-09-30 浙江大学 多人高空电力协同作业的智能ar眼动交互方法和系统
CN115802159A (zh) * 2023-02-01 2023-03-14 北京蓝色星际科技股份有限公司 一种信息显示方法、装置、电子设备及存储介质
CN116051647A (zh) * 2022-08-08 2023-05-02 荣耀终端有限公司 一种相机标定方法和电子设备
CN116222964A (zh) * 2022-11-29 2023-06-06 浙江炽云科技有限公司 一种成像性能测试方法、装置、设备及存储介质
CN117073988A (zh) * 2023-08-18 2023-11-17 交通运输部公路科学研究所 抬头显示虚像距离测量系统及方法、电子设备
WO2023226403A1 (fr) * 2022-05-27 2023-11-30 华为技术有限公司 Carte d'étalonnage et dispositif de commande d'étalonnage
CN117252932A (zh) * 2023-09-25 2023-12-19 深圳市德赛西威汽车电子有限公司 一种ar-hud标定方法、系统、设备及汽车
CN117871049A (zh) * 2023-12-29 2024-04-12 芜湖汽车前瞻技术研究院有限公司 测试装置
CN117994343A (zh) * 2023-12-29 2024-05-07 清华大学 确定mr图像中植入电极方位的方法及设备
CN118247360A (zh) * 2024-04-18 2024-06-25 镁佳(北京)科技有限公司 相机内外参的标定方法、装置、设备、存储介质及程序产品
TWI904549B (zh) * 2024-01-15 2025-11-11 大陸商業成光電（深圳）有限公司 Ar-hud立體物件的補償結構及其方法

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113240592A (zh) * 2021-04-14 2021-08-10 重庆利龙科技产业(集团)有限公司 基于ar-hud动态眼位下计算虚像平面的畸变矫正方法
CN113034618A (zh) * 2021-04-20 2021-06-25 延锋伟世通汽车电子有限公司 一种汽车抬头显示器成像距离测量方法及系统
CN113256739B (zh) * 2021-06-28 2021-10-19 所托(杭州)汽车智能设备有限公司 车载bsd摄像头的自标定方法、设备和存储介质
CN114155300B (zh) * 2021-10-29 2025-05-09 重庆利龙中宝智能技术有限公司 一种车载hud系统投影效果检测方法及装置
CN114332235A (zh) * 2021-11-23 2022-04-12 中国航空工业集团公司洛阳电光设备研究所 车载ar-hud的标定方法
CN114663487B (zh) * 2022-03-09 2025-07-15 北京沃东天骏信息技术有限公司 图像标注方法、装置、电子设备及存储介质
CN115423859A (zh) * 2022-08-10 2022-12-02 歌尔光学科技有限公司 双目融合度的测量系统和测量方法
CN117522994A (zh) * 2023-03-07 2024-02-06 湖南锐思华创科技有限公司 一种便携式hud标定与测试方法、系统和存储介质
CN117033862A (zh) * 2023-10-08 2023-11-10 西安道达天际信息技术有限公司 地理坐标转换为ar坐标的转换方法、系统及存储介质
CN119309780B (zh) * 2024-12-17 2025-04-08 南京睿维视科技有限公司 虚像测量方法及测量设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170169612A1 (en) * 2015-12-15 2017-06-15 N.S. International, LTD Augmented reality alignment system and method
CN108989794A (zh) * 2018-08-01 2018-12-11 上海玮舟微电子科技有限公司 基于抬头显示系统的虚像信息测量方法及系统
CN109859155A (zh) * 2017-11-30 2019-06-07 京东方科技集团股份有限公司 影像畸变检测方法和系统
CN109884793A (zh) * 2017-12-06 2019-06-14 三星电子株式会社 用于估计虚拟屏幕的参数的方法和设备
CN111147834A (zh) * 2019-12-31 2020-05-12 深圳疆程技术有限公司 一种基于增强现实抬头显示的虚拟图像标定方法

Family Cites Families (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP2608153A1 (fr) * 2011-12-21 2013-06-26 Harman Becker Automotive Systems GmbH Procédé et système pour jouer une jeu avec réalité augmentée dans un véhicule à moteur
JP6040564B2 (ja) * 2012-05-08 2016-12-07 ソニー株式会社 画像処理装置、投影制御方法及びプログラム
KR101484170B1 (ko) * 2013-05-06 2015-01-21 주식회사 이미지넥스트 Hud 영상 평가시스템 및 그 평가방법
JP2017157093A (ja) * 2016-03-03 2017-09-07 矢崎総業株式会社 車両用表示装置
CN106127714B (zh) * 2016-07-01 2019-08-20 南京睿悦信息技术有限公司 一种虚拟现实头戴显示器设备畸变参数的测量方法
JP6692939B2 (ja) * 2017-02-15 2020-05-13 マクセル株式会社 ヘッドアップディスプレイ装置
US11181743B2 (en) * 2017-11-14 2021-11-23 Maxell, Ltd. Head up display apparatus and display control method thereof
CN108399640A (zh) * 2018-03-07 2018-08-14 中国工程物理研究院机械制造工艺研究所 一种基于摄像机标定的反射镜相对位姿测量方法
CN207894591U (zh) * 2018-03-12 2018-09-21 福耀集团（上海）汽车玻璃有限公司 一种hud前挡玻璃检测设备
CN110874135B (zh) * 2018-09-03 2021-12-21 广东虚拟现实科技有限公司 光学畸变的校正方法、装置、终端设备及存储介质
CN109472829B (zh) * 2018-09-04 2022-10-21 顺丰科技有限公司 一种物体定位方法、装置、设备和存储介质
KR20200057929A (ko) * 2018-11-19 2020-05-27 주식회사 스튜디오매크로그래프 캘리브레이트된 카메라들에 의해 캡쳐된 스테레오 영상들의 렉티피케이션 방법과 컴퓨터 프로그램
CN109712194B (zh) * 2018-12-10 2021-09-24 深圳开阳电子股份有限公司 车载环视系统及其立体标定方法和计算机可读存储介质
DE102019202512A1 (de) * 2019-01-30 2020-07-30 Siemens Aktiengesellschaft Verfahren und Anordnung zur Ausgabe eines HUD auf einem HMD
CN111508027B (zh) * 2019-01-31 2023-10-20 杭州海康威视数字技术股份有限公司 摄像机外参标定的方法和装置
CN111476104B (zh) * 2020-03-17 2022-07-01 重庆邮电大学 动态眼位下ar-hud图像畸变矫正方法、装置、系统
CN111443490B (zh) * 2020-04-15 2022-11-18 杭州赶梦科技有限公司 一种ar hud的虚像显示区域调节方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170169612A1 (en) * 2015-12-15 2017-06-15 N.S. International, LTD Augmented reality alignment system and method
CN109859155A (zh) * 2017-11-30 2019-06-07 京东方科技集团股份有限公司 影像畸变检测方法和系统
CN109884793A (zh) * 2017-12-06 2019-06-14 三星电子株式会社 用于估计虚拟屏幕的参数的方法和设备
CN108989794A (zh) * 2018-08-01 2018-12-11 上海玮舟微电子科技有限公司 基于抬头显示系统的虚像信息测量方法及系统
CN111147834A (zh) * 2019-12-31 2020-05-12 深圳疆程技术有限公司 一种基于增强现实抬头显示的虚拟图像标定方法

Cited By (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023226403A1 (fr) * 2022-05-27 2023-11-30 华为技术有限公司 Carte d'étalonnage et dispositif de commande d'étalonnage
CN115131529A (zh) * 2022-06-15 2022-09-30 浙江大学 多人高空电力协同作业的智能ar眼动交互方法和系统
CN116051647A (zh) * 2022-08-08 2023-05-02 荣耀终端有限公司 一种相机标定方法和电子设备
CN116222964A (zh) * 2022-11-29 2023-06-06 浙江炽云科技有限公司 一种成像性能测试方法、装置、设备及存储介质
CN115802159B (zh) * 2023-02-01 2023-04-28 北京蓝色星际科技股份有限公司 一种信息显示方法、装置、电子设备及存储介质
CN115802159A (zh) * 2023-02-01 2023-03-14 北京蓝色星际科技股份有限公司 一种信息显示方法、装置、电子设备及存储介质
CN117073988A (zh) * 2023-08-18 2023-11-17 交通运输部公路科学研究所 抬头显示虚像距离测量系统及方法、电子设备
CN117073988B (zh) * 2023-08-18 2024-06-04 交通运输部公路科学研究所 抬头显示虚像距离测量系统及方法、电子设备
CN117252932A (zh) * 2023-09-25 2023-12-19 深圳市德赛西威汽车电子有限公司 一种ar-hud标定方法、系统、设备及汽车
CN117871049A (zh) * 2023-12-29 2024-04-12 芜湖汽车前瞻技术研究院有限公司 测试装置
CN117994343A (zh) * 2023-12-29 2024-05-07 清华大学 确定mr图像中植入电极方位的方法及设备
TWI904549B (zh) * 2024-01-15 2025-11-11 大陸商業成光電（深圳）有限公司 Ar-hud立體物件的補償結構及其方法
CN118247360A (zh) * 2024-04-18 2024-06-25 镁佳(北京)科技有限公司 相机内外参的标定方法、装置、设备、存储介质及程序产品

Also Published As

Publication number Publication date
CN112655024B (zh) 2022-04-22
CN112655024A (zh) 2021-04-13

Similar Documents

Publication Publication Date Title
CN112655024B (zh) 一种图像标定方法及装置
JP5455124B2 (ja) カメラ姿勢パラメータ推定装置
US9858639B2 (en) Imaging surface modeling for camera modeling and virtual view synthesis
CN107113376B (zh) 一种图像处理方法、装置及摄像机
US20170134713A1 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
US20170127045A1 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
EP4343408A1 (fr) Procédés et appareils d'imagerie d'affichage tête haute de réalité augmentée, dispositifs et supports de stockage
WO2022126430A1 (fr) Procédé, appareil et système de mise au point auxiliaire
WO2021104308A1 (fr) Procédé de mesure de profondeur panoramique, caméra à œil de poisson à quatre yeux et caméra fisheye binoculaire
WO2018110264A1 (fr) Dispositif d'imagerie, et procédé d'imagerie
JP4679293B2 (ja) 車載パノラマカメラシステム
CN109945840B (zh) 三维影像摄取方法及系统
JP2011087319A (ja) 車載パノラマカメラシステム
CN118071603A (zh) 空间角度信息交互的光场图像超分辨率方法、装置及介质
US12271993B2 (en) Method for forming an image of an object, computer program product and image forming system for carrying out the method
TWI793584B (zh) 自動泊車建圖與定位的系統及其方法
CN112017138B (zh) 一种基于场景三维结构的图像拼接方法
KR20240066218A (ko) 장면 뷰 이미지의 생성 장치 및 방법
CN212163540U (zh) 全向立体视觉的摄像机配置系统
JP6788472B2 (ja) 要素画像生成装置およびそのプログラム
CN115409973B (zh) 增强现实抬头显示成像方法、装置、设备以及存储介质
KR102762198B1 (ko) 자동차 내부 카메라를 이용한 hud 캘리브레이션 방법 및 장치
CN115409973A (zh) 增强现实抬头显示成像方法、装置、设备以及存储介质
JP2010004227A (ja) 撮像装置
Corke Image Formation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20959264

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20959264

Country of ref document: EP

Kind code of ref document: A1