CN101673473A - Omni-directional vision parking auxiliary device based on DSP and method for generating Omni-directional vision image - Google Patents

Abstract

The invention discloses a DSP-based panoramic vision parking assistance device and a method for generating panoramic vision images. The device adopts a wide-angle lens video collector, an image synchronization processor, a DSP processor, a storage module, a logic controller, and an output frame cache memory, video encoder and image display, the video collectors are respectively fixedly installed on the front, rear, left and right outer surfaces of the vehicle body, the storage module includes flash memory, DDR2 memory, and the flash memory and DDR2 memory are connected with the DSP processor respectively connected, the video collector, image synchronization processor, DSP processor, output frame buffer, video encoder and image display are connected in sequence, and the image synchronization processor, DSP processor, output frame buffer and video encoder are respectively connected with Logic controller connection. The invention has the advantage of being able to provide the driver with the panoramic image of the car needed for parking in real time, thereby eliminating the visual blind spots around the car and helping the driver to park more accurately.

Description

DSP-based panoramic vision parking assistance device and method for generating panoramic vision images

technical field

The invention relates to a visual parking assistance device, belonging to the fields of image processing and computer vision.

Background technique

When human society develops to use automobiles as the main means of transportation, there are higher standards for driving safety and comfort. Among them, the reversing radar appeared first. Early reversing radar mostly used audio as the main auxiliary means. The sensors installed on the rear bumper of the vehicle detected the rear objects of the vehicle, and reminded the driver of the situation behind the vehicle through audio with different rhythms. The reversing radar relies on the echo to detect the distance and prompts through the sound of different frequencies, but the sound prompt is obviously not as intuitive as the vision, and there will inevitably be errors in the judgment of the sound; video surveillance system. This system makes the situation behind the car more intuitive and visible when reversing, and it is one of the very practical configurations for reversing safety. When the reverse gear is engaged, the system will automatically connect to the high-definition camera located at the rear of the vehicle to clearly display the situation behind the vehicle on the LCD screen, and synthesize the vehicle driving route corresponding to the current rotation angle of the steering wheel through the computer on the display , so as to accurately and intuitively grasp the situation behind the car. Obviously, the reversing image monitoring system is more intuitive and practical than the reversing radar. However, only installing a reversing camera at the rear of the car can only cover a limited area around the car, and the visual blind spots on both sides and in front of the car undoubtedly increase the hidden danger of safe driving, not only making it difficult for the driver to know whether there are obstacles, but also improving the safety of passengers. and pedestrian danger.

Contents of the invention

The object of the present invention is to provide a DSP-based panoramic vision parking assistance device and a method for generating panoramic vision images, which can process video images collected in scenes around the car in real time to generate virtual panoramic images.

In order to realize the object of the present invention, the technical solution adopted by the present invention is: the panoramic vision parking aid device based on DSP mainly comprises the video acquisition device that adopts wide-angle lens, image synchronous processor, DSP processor, storage module, logic controller, Output frame buffer, video encoder and image display, the video collector is fixedly installed on the front, rear, left and right outer surfaces of the vehicle body respectively, the storage module includes flash memory, DDR2 memory, and flash memory and DDR2 memory are connected with DSP respectively The processor is connected, and the video acquisition device, the image synchronization processor, the DSP processor, the output frame buffer, the video encoder and the image display are connected in sequence, and the image synchronization processor, the DSP processor, the output frame buffer and the video encoder Connect with the logic controller respectively.

Further, the image synchronization processor of the present invention includes a video decoder and a frame buffer memory, and the video collector, frame buffer memory and logic controller are respectively connected to the video decoder, and the logic controller and the DSP processor are respectively Connect to framebuffer memory.

The method for generating a virtual panoramic image using the panoramic vision parking assistance device of the present invention mainly includes the following steps:

(1) The video collector collects a frame of image, after being processed by the image synchronization processor, the DSP processor stores the image in the DDR2 memory;

(2) the DSP processor reads the virtual panoramic image lookup list pre-stored in the flash memory into the DSP processor;

(3) DSP processor according to the parameter in the search list, after taking out corresponding pixel point from DDR2 internal memory successively, described pixel point is sent to output frame buffer;

(4) After the DSP processor in step (3) sends all corresponding pixels to the output frame buffer according to the parameters in the lookup list, loop to step (1) to start collecting a new frame of image, and the logic controller will simultaneously The content stored in the output frame buffer is sent to the video encoder, and after being encoded by the video encoder, the virtual panoramic image is displayed by the image display.

Further, the obtaining of the virtual panoramic image lookup list of the present invention includes the following steps:

1) Obtain the internal parameters K and distortion coefficient k _{i of} the wide-angle lens used by _each video collector installed on the front, rear, _left , and _right outer surfaces of the vehicle body respectively. k ₅ ], where k ₁ , k ₂ , k ₃ are radial distortion coefficients, k ₄ , k ₅ are tangential distortion coefficients;

2) Park the car on a flat ground, place grid markers in the overlapping area of the viewing angles of adjacent video collectors, and photograph the flat ground by each of the video collectors to obtain a group of original images;

3) the body part in the original image obtained in step 2) is removed and only the flat ground part is reserved, and the position parameter of each pixel coordinate point after the original image is de-distorted is obtained according to the following relational expression, thereby obtaining the corrected image;

x′＝x+(xx ₀ )[k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶ +k ₄ (r ² /x+2x)

y′=y+(yy ₀ )[k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶ +k ₄ (r ² /y+2y)

$\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="2"\; label="y"\; filenames="A2009101531730012H2.png"\; state="\{\{state\}\}">y</figure-callout></span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

Among them, (x', y', 1) is the coordinate of the pixel after dedistortion; (x, y, 1) is the coordinate of the pixel point of the original image; (x ₀ , y ₀ , 1) the coordinate of the lens projection center;

4) Set the virtual viewpoint above the roof of the car, multiply the corrected images by the homography matrix of the virtual viewpoint relative to the position of each video collector, and obtain the corrected image with the virtual viewpoint as the coordinate origin. The top view of each image of ;

5) by translating the top views, the lines of the images collected by the adjacent video collectors for the same grid marker are spliced to obtain a complete grid marker image, thereby obtaining a virtual panoramic image, Record the original image to which each pixel in the virtual panoramic image belongs;

6) For each pixel of the virtual panoramic image, according to the original image to which each pixel in the virtual panoramic image obtained in step 5) belongs, the inverse operation is performed in steps 4) and 3) successively to obtain The corresponding relationship between each pixel point of the virtual visual panoramic image and the pixel point of the image collected by the video collector.

Compared with prior art, the beneficial effect of the present invention is:

The present invention proposes a DSP-based panoramic vision parking assistance device and a method for generating panoramic visual images, which can provide the driver with the panoramic image of the car required for parking in real time, thereby eliminating the visual blind spots around the car to help drive more precise parking for the driver; the DSP-based panoramic vision parking aid device of the present invention includes a video collector, an image synchronization processor, a DSP processor, a storage module, a logic controller, an output frame buffer, a video encoder and an image display ; The process of generating the virtual panoramic image is mainly obtained by addressing the DSP processor according to the parameters of the virtual panoramic image lookup list stored in the flash memory in advance, so the calculation complexity is low and can meet the real-time requirement.

Description of drawings

Fig. 1 is a schematic block diagram of the structure of the device of the present invention.

Fig. 2 is a schematic diagram of the installation position and shooting area of the video collector of the present invention.

Fig. 3 is a diagram of the internal relationship of the image synchronization processor of the present invention.

Fig. 4 is a schematic diagram of the placement position of the lattice marker of the present invention.

Fig. 5 is a schematic diagram of the image correction effect in the present invention.

Fig. 6 is a flow chart of generating a panoramic visual image by the device.

Fig. 7 is a schematic diagram of hardware connection of an embodiment of the device of the present invention.

Detailed ways

As an embodiment of the present invention, the device of the present invention takes the TMS320DM643 processor of TI (Texas instruments incorporated) company as the core, and TMS320DM643 is a new generation of DSP processor specially oriented to multimedia applications released by TI company. It has been widely used in the multimedia field with a dedicated video port and rich expansion interfaces.

Fig. 1 is a schematic block diagram of the structure of the device of the present invention. As shown in Figure 1, it includes a video collector, image synchronization processor, DSP processor, storage module, logic controller, output frame buffer, video encoder and image display. The video collectors are respectively fixedly installed on the front, rear, left and right outer surfaces of the vehicle body; the lens of the video collector adopts a wide-angle lens to respectively collect real-time pictures in four areas of the front, rear, left and right of the vehicle body, and can make two There is a certain overlapping area between the viewing angles of two adjacent video collectors. The video collector can be installed on the upper part of the front radiator grille, the upper end of the rear cover and the lower end of the left and right exterior mirrors, as shown in Figure 2, where C1-C4 are video collectors, point P is a virtual viewpoint, and adjacent Four overlapping regions of video grabber perspective1. The image signal collected by the video collector can be an analog video signal or a digital image signal, and the video collector transmits the collected image signal to the image synchronization processor. When the collected signal is an analog video signal, the image synchronization processor connected to the video collector includes a video decoder and a frame buffer memory (FIFO), and the connection is as shown in Figure 3; at this time, the video decoder converts the analog video signal into The digital signals that can be processed by the DSP processor are stored in the frame buffer memory (FIFO). When the collected signal is a digital signal, the video collector is directly connected to the frame buffer memory (FIFO). The image synchronization processor is connected with the DSP processor, and the DSP processor reads images from the frame buffer memory of the image synchronization processor under the control of the logic controller CPLD, and stores the obtained images in the DDR2 memory of the storage module. The storage module connected to the DSP processor includes flash memory and DDR2 memory. The flash memory is used to store the program code of the DSP processor for loading into the program space in the chip when the power is turned on; in addition, the virtual panoramic image search is also stored in the flash memory. The list and the top view image of the car body are used to generate a virtual panoramic image; the DDR2 memory is mainly used to store the digital signal received from the image synchronization processor. The DSP processor is also connected to the output frame buffer. The main function of the output frame buffer is to receive and temporarily store the virtual panoramic image transmitted by the DSP processor. The output frame buffer is connected with the video encoder, and they convert the digital image signal into an analog video signal under the control of the logic controller CPLD, and the virtual panoramic image is displayed by the image display.

Wherein the virtual panorama image search list stored in the flash memory can be obtained by the following steps:

1) Obtain the internal parameters K and distortion coefficient k _{i of} the wide-angle lens used by _each video collector installed on the front, rear, _left , and _right outer surfaces of the vehicle body respectively. k ₅ ], where k ₁ , k ₂ , k ₃ are radial distortion coefficients, k ₄ , k ₅ are tangential distortion coefficients;

ZHANG, ZY: A flexible new technique for camera calibration. IEEE Transaction on Pattern Analysis and Machine Intelligence 22(11), 1330-1334(2000) discloses a calibration method of lens parameters, which can be used to obtain the internal parameters K of the lens and distortion coefficient _ki = [k ₁ , k ₂ , k ₃ , k ₄ , k ₅ ]. The present invention can also use this calibration method to obtain the internal parameters K and distortion coefficient k _i =[k ₁ , k ₂ , k ₃ , k ₄ , k ₅ ].

2) The car is parked on a flat ground, and grid-shaped markers 2 are respectively placed in four overlapping areas 1 of each adjacent video collector viewing angle as shown in Figure 2; as shown in Figure 4, the grid-shaped markers The object 2 has a checkerboard-like grid, and the flat ground is photographed by each of the video collectors to obtain a group of original images I _i (i=1, 2, 3, 4);

3) Remove the body part in the original image I _i obtained in step 2) and only keep the flat ground part, and obtain the position parameter of each pixel coordinate point after dedistorting the original image I _i according to the following relational expression, so as to obtain the correction After the image U _i (i=1, 2, 3, 4), the correction effect is as shown in Figure 5;

x′＝x+(xx ₀ )[k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶ +k ₄ (r ² /x+2x)

y′=y+(yy ₀ )[k ₁ r ² +k ₂ r ⁴ +k ₃ r ⁶ +k ₄ (r ² /y+2y)

$\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; x</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; the\; <figure-callout\; id="2"\; label="y"\; filenames="A2009101531730012H2.png"\; state="\{\{state\}\}">y</figure-callout></span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

Among them, (x', y', 1) is the coordinate of the pixel after dedistortion; (x, y, 1) is the coordinate of the pixel point of the original image; (x ₀ , y ₀ , 1) the coordinate of the lens projection center;

4) Set the virtual viewpoint above the roof of the car, and multiply the rectified images U _i by the homography matrix H _i (i=1, 2, 3, 4 ), correspondingly obtain the top view F _i (i=1, 2, 3, 4) of each image U _i corrected with the virtual viewpoint as the coordinate origin;

5) By translating the top view F _i , splicing the lines of the images collected by the adjacent video collectors for the same grid marker to obtain a complete grid marker image, thereby obtaining a virtual panoramic image , and according to the stitching seam position, record the original image number i to which each pixel in the virtual panoramic image belongs;

6) For each pixel of the virtual panoramic image, perform inverse operations in steps 4) and 3) sequentially, that is, for each pixel of the virtual panoramic image (x ₀ ″, y ₀ ″, 1), in combination with step 5 ) The original image number i to which each pixel of the obtained virtual panoramic image belongs is multiplied by the H _i ^-1 matrix, (H _i ^-1 is the inverse matrix of H _i ) to obtain the result (x ₀ "', y ₀ "', 1) into the relational expression described in step 3), let (x', y', 1)=(x ₀ "', y ₀ "', 1), and calculate (x, y, 1) The corresponding relationship between each pixel point of the virtual vision panoramic image and the pixel point of the original image collected by the video collector is thus obtained.

Since the car body part is removed in step 3), there is a blank area in the virtual panoramic image obtained in step 5), and the virtual car body top view image can be used as filling, and the virtual car body top view image can be stored in advance. memory modules, such as flash memory. In the above process, the pixels of the non-virtual car body part of the virtual panoramic image are all obtained by a certain pixel in the original image I _i collected by the video collector through a series of space transformations, and the The content, i.e. its brightness or color information, is independent, and for a lens with a fixed position, its internal parameters and distortion coefficients, as well as the homography matrix for a certain position are constant. Therefore, we record the point correspondence between the original image and the virtual panoramic image and save it in a list format. When the scene captured by the video collector changes, the point correspondence between the original image and the virtual panoramic image obtained above is still used for processing. , to generate a virtual panoramic image. Store this list in flash memory. Each pixel parameter in this list format can be expressed as (n, x, y), where n=1, 2, 3, 4, 5; 1 represents I ₁ , 2 represents I ₂ , 3 represents I ₃ , and 4 represents I ₄ and 5 represent the top view image of the virtual car body, and x and y represent the abscissa and ordinate of the original image corresponding to the point at each location. For example, the first pixel point parameter is (1, 4, 5), which means that the pixel point in the first row and first column of the virtual panoramic image corresponds to the pixel point whose abscissa is 4 and the ordinate is 5 in the original image _I1 .

The virtual panoramic image search column takes the upper left corner of the virtual panoramic image as the starting pixel point (stored in the first position of the list), and arranges from left to right and from top to bottom. For example, the parameters of the pixel points in the third row and the fourth column of the virtual panoramic image are recorded in the position (2×total number of columns+4) in the list.

The working flow diagram of the device of the present invention is as shown in Figure 6, and the method for generating a virtual panoramic image comprises the following steps:

(1) The video collector collects a frame of image, (when the image collected by the video collector is an analog video signal image, it also needs to be A/D converted by the video decoder in the image synchronization processor to convert it into a digital image signal) Stored in the frame buffer memory, when each frame buffer memory is full of a frame of image, the logic controller opens the frame buffer memory output enable terminal, and the digital image signal is read in synchronously by the DSP processor, and then the DSP processor converts the image Stored in DDR2 memory;

(2) the DSP processor reads the virtual panoramic image lookup list pre-stored in the flash memory into the DSP processor;

(3) DSP processor according to the parameter in the search list, after taking out corresponding pixel point from DDR2 internal memory successively, described pixel point is sent to output frame buffer;

(4) After the DSP processor in step (3) sends all corresponding pixels to the output frame buffer according to the parameters in the lookup list, loop to step (1) to start collecting a new frame of image, and the logic controller will simultaneously The content stored in the output frame buffer is sent to the video encoder, and after being encoded by the video encoder, the virtual panoramic image is displayed by the image display.

Specifically, taking the analog video signal collected by the video collector as an example, when the video collector is started at a certain moment, the four video collectors start to collect images and transmit the collected analog video signals to the image synchronization processor. The video decoding chip in the image synchronization processor can choose Philips' 9-bit video decoder SAA7115. The SAA7115 is programmed on its I2C bus through a logic controller (CPLD). The SAA7115 can interface with all major composite video standards, such as NTSC, PAL and SECAM, which can be programmed through the decoder's internal registers. Since this device has high requirements for real-time performance, the configuration of SAA7115 is: adopt 4:2:2 YUV signal mechanism, and only take Y (brightness) signal. Frame buffer memory and output frame buffer can use AVERLOGIC's AL4V8M440 chip, the storage space is 8M (1024k x8-bit), has independent input and output ports, and can accept different I/O read and write rates; the maximum read and write speed up to 80MHz. The 8-bit parallel output port Y[9:2] of SAA7115 is connected with the data input port DI[7:0] of the frame buffer memory AL4V8M440.

The line and field synchronization signal output by SAA7115 is first transmitted to the logic controller (CPLD) as a frame synchronization logic control signal. When the image starts to be collected, 4 asynchronous field synchronous signals are input into the CPLD, and then connected to the WE port (write enable port) of the corresponding frame buffer memory AL4V8M440 through the CPLD, and a frame signal is input. The four AL4V8M440 data output ports DO[7:0] are respectively connected to the respective two Channels of VP1 and VP2 of the DSP. After all 4 AL4V8M440s have stored a frame of image, CPLD simultaneously sets the read enable signal RE of 4 AL4V8M440 to allow reading, allowing the VP port of the DSP processor to read a frame of image from AL4V8M440. The image data is first written into the first-in-first-out register with a size of 5120 bytes in the DSP. When the DSP is applied in dual-channel mode, the first-in-first-out register is automatically split into two channels with a size of 2560 bytes. Each channel can An Enhanced Direct Memory Access (ie, EDMA) is set for moving. EDMA stores the images collected by 4 video capture devices in DDR2 memory.

After each frame of image collected by the four video collectors is stored in the DDR2 memory, the DSP processor reads the virtual panoramic image lookup list from the flash memory, and the DSP processor takes out the corresponding image from the DDR2 memory according to the parameters in the lookup list. Pixels, and then send the pixels to the output frame buffer; when the DSP processor sends all the corresponding pixels to the output frame buffer according to the parameters in the lookup list, the timing signal of the output frame buffer and the video encoding chip are generated by the CPLD Line and field synchronous signal, the content stored in the output frame buffer is transmitted to a video encoder (such as NXP's video encoding chip SAA7105), converted into a video signal, and transmitted to the LCD screen by the video line to display a virtual panoramic image. At the same time, the video collector starts to collect a new frame of image. The schematic diagram of the hardware connection in this example is shown in Figure 7.

Since the process of generating the virtual panoramic image by the device is mainly realized through DSP addressing, the calculation complexity is low and the real-time requirement can be met.

Claims (4)

1. panoramic vision parking assist apparatus based on DSP, it is characterized in that: it comprises the video collector that adopts wide-angle lens, the image synchronization processor, dsp processor, memory module, logic controller, the output frame buffer memory, video encoder and image display, described video collector be fixedly mounted on respectively vehicle body before, after, a left side, on the right outside surface, described memory module comprises flash memory, the DDR2 internal memory, flash memory is connected with dsp processor respectively with the DDR2 internal memory, described video collector, the image synchronization processor, dsp processor, the output frame buffer memory, video encoder is connected successively with image display, described image synchronization processor, dsp processor, the output frame buffer memory is connected with logic controller respectively with video encoder.

2. the panoramic vision parking assist apparatus based on DSP according to claim 1, it is characterized in that: described image synchronization processor comprises Video Decoder and frame buffer memory, described video collector, frame buffer memory and logic controller are connected with Video Decoder respectively, and described logic controller is connected with frame buffer memory respectively with dsp processor.

3. a panoramic vision parking assist apparatus that uses claim 1 generates the method for virtual panoramic image, it is characterized in that may further comprise the steps:

(1) video collector is gathered a two field picture, after the image synchronization processor processing, dsp processor with image storage in the DDR2 internal memory;

(2) dsp processor will be pre-stored in virtual panoramic image in the flash memory and search tabulation and be read in the dsp processor;

(3) dsp processor behind the pixel of taking-up correspondence, sends to the output frame buffer memory with described pixel successively according to the parameter of searching in the tabulation from the DDR2 internal memory;

(4) after dsp processor in the step (3) sends to the output frame buffer memory according to the pixels of searching the whole correspondences of parameter general in the tabulation, be circulated to step (1) and begin to gather a new two field picture, logic controller sends to video encoder with content stored in the output frame buffer memory simultaneously, through after the video encoder encodes, show the virtual panoramic image by image display.

4. panoramic vision parking assist apparatus according to claim 3 generates the method for panoramic vision image, it is characterized in that the acquisition that described virtual panoramic image is searched tabulation may further comprise the steps:

1) obtains the inner parameter K and the percentage distortion k of the used wide-angle lens of each video collector on the front, rear, left and right outside surface that is installed in vehicle body respectively _i=[k ₁, k ₂, k ₃, k ₄, k ₅], k wherein ₁, k ₂, k ₃Be radial distortion coefficient, k ₄, k ₅Be tangential percentage distortion;

2) with automobile parking on smooth ground, in the overlapping region at adjacent video collector visual angle, place the trellis label, by described each video collector smooth ground is taken, obtain one group of original image;

3) with step 2) body portion in the original image that obtains removes and only keeps face portion flatly, obtains the location parameter that described original image removes each the pixel coordinate point after the distortion according to following relational expression, thus the image after obtaining correcting;

x′＝x+(x-x ₀)[k ₁r ²+k ₂r ⁴+k ₃r ⁶+k ₄(r ²/x+2x)

y′＝y+(y-y ₀)[k ₁r ²+k ₂r ⁴+k ₃r ⁶+k ₄(r ²/y+2y)

$r=\sqrt{x^2+y^2}$

Wherein, (x ', y ', 1) removes coordinate after the distortion for pixel; (x, y, 1) is the original image pixels point coordinate; (x ₀, y ₀, 1) and camera lens projection centre coordinate;

4) set the virtual view of car roof top, with each image after correcting multiply by respectively this virtual view with respect to each video collector position homography matrix, corresponding obtaining with this virtual view is the vertical view of each image after the rectification of true origin;

5) by described each vertical view of translation, the lines of the image that described adjacent video collector is collected at same trellis label splice and obtain complete trellis label image, obtain the virtual panoramic image thus, the original image that each pixel was subordinate in record virtual panoramic image this moment;

6) for each pixel of described virtual panoramic image, according to the original image that each pixel was subordinate in the virtual panoramic image that obtains in the step 5), successively set by step 4), 3) carry out inverse operation, obtain each pixel of virtual vision panoramic picture of this moment and the corresponding relation of video collector institute images acquired pixel.

Images