JPH08333086A - Picked-up image processing device - Google Patents

Abstract

(57) [Summary] [Purpose] The main object of the present invention is to enable accurate tracking of the position of a suspended load even in an environment where the brightness changes. Structure: Each time pattern matching is performed, the brightness data of the pixel of the pattern image is set so that the brightness of the pixel of the pattern image becomes the brightness of the pixel of the captured image when the captured image and the pattern image are matched. Will be updated sequentially.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for picking up an image of a suspended load suspended on the tip of a boom, and processing the captured image to perform a predetermined process such as tracking the position of the suspended load. .

[0002]

2. Description of the Related Art Stopping the swinging of a suspended load suspended by a rope at the tip of a boom of a mobile crane such as a rough terrain crane ensures work safety. It is important for reasons such as

Conventionally, the stop control of the shake of the load is performed by using the position of the load (the deflection angle) obtained by the image processing as a feedback amount, as seen in Japanese Patent Laid-Open No. 4-89795. The position of the suspended load (the swing angle) is feedback-controlled so that the load stops (stops vertically below the boom).

However, when such feedback control is applied to a mobile crane, the control loop operates even when the operation lever is in the neutral state, and when the load shakes for some reason, the operation lever is in the neutral state. The car body moves.

Further, in a mobile crane that requires operability, particularly during fine operation (when the operation lever opening is small),
The smaller the effect of feedback, the better.

The present invention has been made in view of the above circumstances, and it is a first object of the present invention to perform swing stop control of a suspended load without using feedback control. Further, in the case of performing the swing stop control of the suspended load as described above, as a premise, it is necessary to sequentially detect the position of the suspended load by image processing and track the position of the suspended load.

Here, the suspended load suspended at the tip of the boom is imaged, and the captured image and the pattern image showing the suspended load are sequentially subjected to pattern matching.
A tracking system has been already known that is designed to sequentially detect the position of a suspended load on a captured image and perform tracking of the suspended load.

An example of this type of tracking system is the "MEP system" of the Inoue Laboratory of the University of Tokyo.

In this "MEP system", an image of a local region of a captured image and a reference pattern image stored in advance (for example, as an indication of a suspended load, without using the entire region of the captured image, When a target mark provided in the vicinity of the hook of the suspended load is used, the pattern matching is performed at high speed by a correlation calculation by making a match with the target mark).

This "MEP system" has no particular problem in tracking accuracy of suspended loads when used in an environment such as indoors where the illuminance is approximately constant, but when used outdoors, the illuminance is The fluctuation may deteriorate the accuracy of tracking of the suspended load. Therefore, the present invention makes it possible to accurately perform pattern matching between the picked-up image of the suspended load and the pattern image of the suspended load even in an environment where the brightness variation occurs, and thus the position of the suspended load can be tracked. A second object is to enable accurate processing.

Further, in order to perform the above-mentioned pattern matching, as a premise, a suspended load (a target mark indicating a suspended load) suspended at the end of the boom is imaged by a television camera attached to the end of the boom. However, it is necessary to make the ratio of the size of the captured image to the size of the captured suspended load (target mark) constant. This is because the size of the suspended load (target mark) in the captured image and the size of the suspended load (target mark) in the pattern image need to match.

However, during the crane work, the length of the rope changes and the distance between the TV camera and the load (target mark) at the tip of the rope usually changes. The size of the suspended load (target mark) also changes constantly.

Here, the "shape" of an object such as a line
When performing image processing as a feature amount, there is no particular problem, but when performing image processing in which the “size” of the object influences the feature amount, the above “size” fluctuations greatly affect the processing accuracy. Influence. In particular, when performing the above-described pattern matching, “size” is a very important feature amount, and therefore the size is changed every time the size of the suspended load (target mark) in the captured image changes. Accordingly, it is necessary to exchange the reference pattern image. However, this causes an increase in processing time and storage capacity, which is not desirable.

By the way, as a conventional technique for adjusting an image picked up by a television camera provided on a crane or the like, there is, for example, Japanese Patent Laid-Open No.
No. 80388 and Japanese Utility Model Laid-Open No. 3-89083, both of which are intended for operator operability and visual field adjustment, and the size of the suspended load imaged by the television camera is adjusted. It is not intended to be constant.

The present invention has been made in view of these circumstances, and even if the rope length changes due to crane work or the like, the ratio of the size of the suspended load to the image picked up by the image pickup means is always a desired constant value. It is a third object of the present invention to make it possible to perform processing such as pattern matching with good work efficiency without increasing the storage capacity.

[0016]

In order to achieve the above first object, in the first aspect of the present invention, an image of a suspended load suspended at the tip of the boom is imaged and the captured image is processed. By detecting the position of the suspended load on the image, and based on the detected position of the suspended load on the image, in a suspended load imaging image processing device for performing shake stop control for stopping the swing of the suspended load. A motion equation describing the relationship between the swing angle and swing angular velocity of the suspended load and the moving acceleration of the boom tip, and the swing angle of the suspended load should start the swing stop control based on the equation of motion. Moving speed change that generates a moving speed change pattern that shows how the moving speed of the boom tip changes from when the swing angle reaches a predetermined swing angle to when the swing angle and the swing angular velocity of the suspended load become zero at the same time. Turn generation means, control start time point detection means for detecting that the swing angle of the suspended load reaches the predetermined swing angle by tracking the position of the suspended load on the image, and the control start time point detection means And a moving speed control means for controlling the moving speed of the boom tip according to the moving speed change pattern generated by the moving speed change pattern generating means from the time when it is detected that the predetermined swing angle is reached. There is.

In order to achieve the second object,
In the second main invention of the present invention, the suspended load suspended on the tip of the boom is imaged, and the captured image and the pattern image indicating the suspended load are sequentially subjected to pattern matching to suspend the suspended image on the captured image. In a picked-up image processing device for a suspended load that sequentially detects the position of the load, the brightness of the pixel of the pattern image becomes the brightness of the pixel of the picked-up image when the picked-up image and the pattern image are matched. As described above, every time the pattern matching is performed, the brightness data of the pixels of the pattern image is sequentially updated.

Further, in order to achieve the third object,
In the third aspect of the present invention, the suspended load suspended at the tip of the boom is imaged by the imaging means, and the ratio between the size of the captured image and the size of the captured suspended load is made constant. In a picked-up image processing apparatus for a suspended load that performs a predetermined process, a rope length detection unit that detects a rope length of the suspended load, and the imaging unit based on the rope length of the suspended load detected by the rope length detection unit. And a control means for changing the zoom amount of the image pickup means so that the ratio of the size of the image picked up in (1) to the size of the picked up load becomes a desired constant value.

[0019]

According to the first aspect of the present invention, the equation of motion describing the relationship between the swing angle and swing angular velocity of the suspended load and the moving acceleration of the boom tip can be obtained. Then, based on this equation of motion, the boom tip from the time when the swing angle of the suspended load reaches a predetermined swing angle at which swing stop control should be started to the time when the swing angle and the swing angular velocity of the suspended load become zero at the same time. A movement speed change pattern indicating a change in the movement speed is generated. Then, by tracking the position of the suspended load on the image, it is detected that the swing angle of the suspended load has reached the predetermined swing angle, and from this time point, the movement speed of the boom tip is changed according to the movement speed change pattern. Is controlled. As a result, the swing of the load eventually stops.

As described above, the position data of the suspended load obtained by the image processing is limited to the minimum use of only detecting the position where the control is started, and thereafter, the position of the suspended load is not fed back and the load swing is performed. Since the stop control is performed, the inconvenience caused by the feedback control can be avoided, and the suspended load can be accurately controlled. Further, according to the second aspect of the invention, the pattern is matched every time the pattern matching is performed so that the brightness of the pixel of the pattern image becomes the brightness of the pixel of the captured image when the captured image and the pattern image are matched. The lightness data of the pixels of the image are sequentially updated.

As described above, even in an environment where the brightness constantly changes, such as outdoors, the brightness of the pixel of the picked-up image of the suspended load and the brightness of the pixel of the pattern image of the suspended load are successively calculated. Since the brightness of the pixels of the pattern image is updated so as to be the same, pattern matching can be performed with high accuracy, and thus the position of the suspended load can be accurately tracked.

According to the third invention, the zoom amount of the image pickup means is changed based on the rope length of the suspended load detected by the rope length detection means, and the size of the image picked up by the image pickup means, and The ratio to the size of the imaged suspended load is set to a desired constant value.

Thus, even if the rope length changes due to crane work or the like, the ratio of the size of the suspended load to the image picked up by the image pickup means is always set to a desired constant value, so that pattern matching or the like is performed. Processing can be performed efficiently and without increasing the storage capacity.

[0024]

Embodiments of the picked-up image processing apparatus for a suspended load according to the present invention will be described below with reference to the drawings.

FIG. 1 shows a mobile crane 1 applied to the embodiment.
It is a figure which shows the structure of.

As shown in FIG. 1, the crane vehicle 1 is roughly composed of an upper swing body 4, a boom 2 and a cab (driver's cab) 3 arranged on the upper swing body 4. In FIG. 1, the crane vehicle 1 is fixed to the ground by an outrigger.

Inside the cab 3, a boom 2, an upper revolving structure 4, and an electric operation lever 18 (see FIG. 2) for operating the traveling mechanism are arranged. Depending on the operation amount of the electric operation lever 18. Thus, the corresponding boom 2, upper swing body 4, and traveling mechanism are driven.

The boom 2 is rotatable in an up-and-down direction A and a swinging direction B which is a swinging direction of the upper swing body 4, and the boom 2 is rotated as follows.

That is, the operating lever 1 for driving the boom 2
A signal indicating the manipulated variable of 8 is applied to the turning and undulating controller 16, and a drive signal is output from this controller 16. The drive signal output from the swing / relief controller 16 is supplied by the swing / relief hydraulic circuit 17 to electric / electric
The hydraulic pressure is converted, and pressure oil having a flow rate according to the drive signal is supplied to the hydraulic cylinder for driving the boom, whereby the boom 2 is rotated.

A rope 5 hangs vertically downward from the tip of the boom 2 via a pulley, and a hook 6 to which a load 7 is hung is arranged at the tip of the rope 5. A ring-shaped target mark 8 (see FIG. 8) when viewed from above is provided on the upper portion of the hook 6.

Here, the rope 5 can be wound by a winch or the like. Further, the boom 2 is configured so that its length can be freely extended / retracted. As a result, the rope length (the length from the boom tip 2a to the hook 6) changes, but the changing rope length is sequentially detected by the rope length sensor 23.

A target mark 8 is attached to the boom tip 2a.
A camera 9 for picking up the image is provided. The camera 9 is mounted on the boom 2 via the camera mounting mechanism 10 that tilts the camera 9 so that the direction in which the camera 9 captures images is always vertically downward even when the boom 2 rotates in the up-and-down direction A. ing. With this camera mounting mechanism section 10, the target mark 8 is always caught within the field of view of the camera 9 regardless of the up-and-down angle of the boom 2.

The image pickup signal picked up by the camera 9 is transmitted to the image processing device 13 in the cab 3 via the electric transmission module 11. Here, the length of the cord that transmits the image pickup signal is adjusted by the cord reel 12.

The cab 3 has an automatic load shake stop button 1
5, the display 14 is provided, and a signal indicating that the automatic load shaking stop button 15 has been turned on is input to the image processing device 13, and the processing result of the image processing device 13 is
Displayed on the display 14.

First Embodiment Next, with reference to FIG. 2 to FIG. 7, the load shake stop control of the suspended load 7 will be described.

FIG. 2 is a block diagram showing the structure of the load shake stop control device of the first embodiment. FIG. 6 is a flowchart showing the procedure of processing performed by the load shake stop control device of FIG.

As shown in step 101 of FIG. 6, first, in the image processing apparatus 13, the load swing position of the suspended load 7 is converted into the load swing angle.

That is, the suspended load 7 of the mobile crane 1 swings in the up-and-down direction A and the turning direction B. Therefore, when this state is imaged by the camera 9, the locus when the suspended load 7 is swinging is projected on the image image plane 25 as indicated by Q (see FIG. 2). In addition, in reality, the trajectory of the target mark 8 is imaged.

The shake in the up-and-down direction A is caused by the image surface 25.
It is captured as a position change in the y direction above. Then, the shake in the turning direction B is captured as a position change in the x direction on the image surface 25.

Of the shakes in both directions, the case where the shake in the turning direction B is stopped will be described below as a representative example.

The position x in the turning direction is detected by the image processing device 13 as shown in the second and third embodiments described later.

The position x in the turning direction is based on the length of the rope (pendulum length) Lp, the ratio of the image and the actual size (angle of view of the camera 9, distance to the subject, etc.).
It is converted to the load deflection angle ψ in the turning direction. As a result, the maximum deflection angle ψmax corresponding to the maximum deflection position xmax of the suspended load 7 can also be obtained.

The turning direction load deflection angle ψ is applied to the controller 16 via the multiplier 19. Similarly, the undulating direction product shake angle θ is added to the controller 16 via the multiplier 20 (step 101). Next, whether or not the operation lever 18 for turning the boom 2 is in the neutral position is determined by detecting the operation stroke amount St of the operation lever 18 (step 10).
2).

As a result, when it is determined that the turning operation lever 18 is not in the neutral position, steps 106 to 1 are executed.
As indicated by 08, control is performed to stop the shake of the load by using the shake angle ψ as a feedback amount.

That is, the detected operation stroke amount S
t is input to the multiplier calculation unit 21, and the operation stroke amount St
A multiplier Ls corresponding to the absolute value of is calculated and output to the multiplier 19 (step 106).

Fs = ψK1Ls (1) where ψ: current swing direction load deflection angle K1: calculation for obtaining feedback amount Fs called gain
And is input to the controller 16 (step 107).

Then, in the controller 16, an operation for obtaining a deviation between the operation stroke amount St which is the current lever command and the feedback amount Fs, Sv = (St−Fs) · K2 (2) where K2: gain is This is executed, and a drive signal corresponding to the deviation Sv is output to the swing / relief hydraulic circuit 17. As a result, the upper swing body 4 (boom 2) is swung so that the swinging of the load is stopped.

Here, from the above formula (1), the operating lever 1
It can be seen that the smaller the opening of 8 is, the smaller the effect of the feedback of the load swing angle ψ is.

Therefore, the effect that the operability of the operating lever is improved can be obtained (step 108).

On the other hand, when it is determined that the turning operation lever 18 is in the neutral position (determination Y in step 102).
ES), as shown in steps 103 to 105, the swing of the load is stopped by the open loop control.

That is, first, it is determined whether or not the automatic goods shake stop button 15 is currently turned on (step 103).

When the automatic load shake stop button 15 is not pressed and is in the off state, the following load shake stop control by the open loop control is not performed.

On the other hand, when the automatic shake-stop button 15 is pressed and is in the ON state, the following shake-stop control by open loop control is executed.

This control obtains a moving speed change pattern in the turning direction of the boom 2 for stopping the swing of the suspended load 7, and drives the swinging operation of the boom 2 (that is, the upper turning motion) at a speed according to the moving speed change pattern. This is performed by driving a hydraulic actuator (for driving the body 4).

FIG. 4 is a physical model showing the pendulum movement of the suspended load 7 in xy coordinates.

Here, m is the weight of the suspended load 7, xs is the position of the boom tip 2a in the swivel direction, x is the position of the suspended load 7 in the swivel direction, y is the vertical position of the suspended load 7, and g is the acceleration of gravity. Further, in the following, "x." Means the first derivative of x, and "x .." means the second derivative of x. Further, (x) 2 means x squared.

Then, the kinetic energy T becomes T = (1/2) m [(xs. + X.) 2+ (y.) 2] ... (3), and the potential energy V is V = -mgLpcos? ) And the position of the suspended load 7 is as follows: x = Lpsin ψ, y = −Lp cos ψ (5)

Therefore, the Lagrangian L is L = TV = (1/2) m [(xs.) 2 + 2xs. ψ. Lpcosψ + (Lp) 2 (ψ) 2] + mgLpcosψ (6) Therefore, ∂ (L) / ∂ (ψ.) = ML pxs. cos ψ + m (Lp) 2 ψ. (7) d / dt (∂L / ∂ψ.) = M (Lp) 2ψ. ． + ML pxs. ． cos ψ-mL pxs. ψ. sin ψ ... (8) ∂L / ∂ψ = -mxs. ψ. Lpsin ψ-mg Lpsin ψ (9) Here, d / dt (∂L / ∂ψ.) − ∂L / ∂ψ = 0 (10). Therefore, ψ. ． + (G / Lp) sin ψ = − (xs ../ Lp) cos ψ ... (11) Here, sin ψ → ψ and cos ψ → 1 can be set, and after all, ψ. ． + (G / Lp) ψ = −xs. ． / Lp (12) is obtained.

According to the equation of motion thus obtained,
Load deflection angle ψ (t), load deflection angular velocity ψ. (T), load deflection angular acceleration ψ. ． (T) is obtained as follows.

Ψ (t) = A (cos√ (g / Lp) t) + B (sin√ (g / Lp) t) −xs. ． / G (13) ψ. (T) = A√ (g / Lp) (sin√ (g / Lpt)) + B√ (g / Lp) × (cos (g / Lpt)) (14) ψ. ． (T) = − Ag / Lp (cos√ (g / Lp) t) −Bg / Lp × (sin√ (g / Lp) t) (15) Here, the load swing angle ψ = 0 and the load swing angular velocity ψ. = Ψ0.
As an initial condition, ψ = 0, ψ. The time t until = 0 is obtained from the above equations (13) to (15).

Then, t = π√ (Lp / g) (16) As a result, the velocity xs. Change pattern PT is required.

That is, the time t is 0 to (π / 2) √
Up to (Lp / g), the upper revolving superstructure 4 has an acceleration of a = xs. ． = Ψ0. √ (gLp) / 2 (0 ≦ t ≦ (π / 2) √ (Lp / g)) (17) The equation of motion at this time is as follows.

Ψ (t) = (a / g) (cos√ (g / Lp) t) + (2a / g) × (sin√ (g / Lp) t) −a / g (18) ψ. (T) =-(a / √ (gLp)) (sin√ (g / Lp) t) + (2a / √ (gLp)) (cos√ (g / Lp) t) (19) ψ. ． (T) = − (a / Lp) (cos√ (g / Lp) t) − (2a / Lp) × (sin√ (g / Lp) t) (20) Further, the time t is (π / 2). ) √ (Lp / g) to π√ (Lp
/ G), acceleration of the upper revolving superstructure 4 is -a = xs. ． = -Ψ0. √ (gLp) / 2 ((π / 2) √ (Lp / g) 0 <t ≦ π√ (Lp / g)) (21) The equation of motion at this time is as follows.

Ψ (t) = − (a / g) (sin√ (g / Lp) t) + a / g (22) ψ. (T) =-(a / √ (gLp)) (cos√ (g / Lp) t) (23) ψ. ． (T) = (a / Lp) (sin√ (g / Lp) t) (24) Now, according to the law of conservation of energy, (m / 2) (Lp ψ0.) 2 = mgLp (1-cos ψmax) (25) ) Holds. From this, the initial load deflection angular velocity ψ0. = √ (2 (g / Lp) (1−cos ψmax)) (26) Since this is substituted into the above equations (17) and (21), acceleration, a = g√ ((1− cos ψ max) / 2) (0 ≦ t ≦ (π / 2) √ (Lp / g)) − a = −g √ ((1−cos ψmax) / 2) ((π / 2) √ (Lp / g) 0 <T ≦ π√ (Lp / g)) (27) is obtained. Here, the maximum shake angle ψmax corresponding to the maximum shake position xmax of the suspended load 7 is detected by the image processing device 13 as described above. The rope length Lp is detected by the rope length sensor 23. Therefore, the above (2
The right side of the equation 7) is uniquely determined, and the speed change pattern PT of the upper swing body 4 is determined (step 104).

FIG. 3 is a graph showing how the position x of the hanging load 7 in the turning direction changes. The position x in the turning direction is detected by the image processing device 13 and is sequentially converted into the swing angle ψ.

Therefore, when x = (xmax + xmin) / 2 is detected, that is, when the swing angle ψ = 0, the speed command (drive signal) according to the speed change pattern PT is generated by the controller 16. And output to the swing / relief hydraulic circuit 17. As a result, the upper swing body 4 (boom 2) is swung with a speed change according to the pattern PT, and eventually the swinging of the suspended load 7 is stopped when the time π√ (Lp / g) has elapsed from the start of control ( Step 10
5; see FIG. 5).

FIG. 7 is a graph showing a simulation result of control by the speed change pattern. As shown in the figure, it can be seen that the load shake X can be made almost zero in a short time from the start of control.

In this embodiment, the control when the suspended load 7 swings in the turning direction has been described, but the control when the suspended load 7 swings in the undulating direction can be similarly performed. .

Further, in the embodiment, the load shake stop control is started when the load shake angle is zero, but the present invention is not limited to this, and the load shake angle at which the control should be started can be set arbitrarily. Is.

By the way, steps 102 and 103 in FIG.
As shown in FIG. 5, when the operation lever 18 is in the neutral position, the control for stopping the shake only when the button 15 is pressed by the operator is performed for the following reason.

That is, even if the operation lever is at the neutral position, if the load shake stop control is automatically performed, the hanging load 7 is shaken against the operator's intention not to work (the operation lever is neutral). The crane vehicle 1 moves to stop. For example, it is assumed that the operating lever is neutral and the suspended load 7 is stopped. At this time, the mobile crane 1 is stationary. However, when the suspended load 7 shakes due to the influence of wind or the like, there is a disadvantage that the crane vehicle 1 moves against the operator's intention in an attempt to stop the shake.

Therefore, the operator's intention is respected, and only when the button 15 for confirming the intention of the shake stop is pressed, the crane vehicle 1 is moved to stop the shake of the load. However, if there is no particular problem, even if the operation lever is in the neutral position, the crane vehicle 1 may be automatically moved to stop the shake of the load.

Second Embodiment Next, a tracking (position tracking) process of the suspended load 7 performed by the image processing apparatus 13 will be described with reference to FIGS. 8 to 15.

FIG. 8 is a block diagram showing the configuration of the apparatus of this second embodiment.

FIG. 9 is a diagram for explaining the pattern matching performed in this embodiment.

That is, as shown in FIG.
A color target mark 8 is prepared as a pattern to be tracked, and this target mark 8 is CC
An image is picked up by the D camera 9, and the image pickup signal is output to the image processing device 13.

In the image processing device 13, each pixel of the picked-up image is represented as lightness data (grayscale value) of a predetermined gradation from white to black. Then, as shown in FIG. 9, the captured image 30
And the reference block RB, which is a reference pattern image showing the target mark 8, are matched and the correlation between them is performed to perform pattern matching (template matching). Then, it is determined that the suspended load 7 is present at the coordinate position of the block B of the captured image 30 when the correlation is highest, and the detected position of the suspended load 7 is output. At this time, as will be described later, the lightness data of each pixel of the reference block RB is taken every time pattern matching is performed, and the picked-up images 30 and 31 at that time
The brightness and the offset are adjusted so that the brightness is updated according to the brightness of each pixel.

As will be described later, the matching between the reference block RB and the picked-up images 30 and 31 is limited to a local range called a search window SW.

In this way, the position of the suspended load 7 is successively detected, and the result is displayed on the display 14.

As a result, the operator
Crane work can be monitored by looking at the display contents of 4. In particular, when combined with the first embodiment described above, the display 14 functions as a monitor during shake stop control.

Next, the tracking processing of the suspended load 7 will be described with reference to the flowchart of FIG.

First, as shown in step 201, the CCD camera 9 captures an image of the target mark 8 and
The reference block RB, which is a template, is loaded into a predetermined memory. In order to obtain an accurate image, it is necessary to keep the mobile crane 1 stationary and the target mark 8 stationary during this loading. Thus, as shown in FIG. 11, the reference block RB indicating the target mark 8 of, for example, 16 × 16 pixels (256 pixels) is acquired (step 201).

Next, a partial image which seems to include the target mark 8 is taken out of the picked-up image, and this partial image is subjected to a normalized correlation search to initialize the reference block RB in the XY coordinate system. Coordinate position (X
1, Y1) is acquired (step 202).

Thus, one reference block RB at the initial coordinate position (X1, Y1) is created (step 2).
03).

Next, pattern matching is performed.

This pattern matching method is shown in FIG.
As shown in, first, in the frame of the previous captured image 30, for example, the search window SW that is a local area image of 48 × 48 pixels is set. This search window SW
The coordinate position of is the same as the coordinate position of the reference block RB. Then, in the next imaged frame 31, the reference block RB is sequentially moved within the set search window SW, and the reference block RB and each candidate block B (a part of the imaged image 31) having the same size as that of the reference block RB are formed. Correlation is performed and the candidate block B having the highest correlation is searched for. Then, when it is determined that the candidate block B having the highest correlation can be searched and the matching can be performed, the moving amount V (VX1,
VY1) is calculated (step 204). The following expression (28) is a function indicating the degree of correlation between R and s, and the fact that the correlation is highest in step 204 is as follows.
It is judged that the value of the function shown in the equation 8) has become the smallest.

[0087] (28) Therefore, the minimum function value at the time of matching is acquired from this equation (step 205).

Thus, the position (X1 + VX1, Y1 + VY1) of the block B when the correlation becomes highest, that is, the matching position is the position where the target mark 8 exists, that is, the position where the suspended load 7 exists. Detected as present. Then, the matching position (X1 + VX1, Y1
+ VY1), set position of search window SW (X1,
Y1) is updated (step 207). In this way, the position of the search window SW is updated, and step 20
The process of 4 is repeatedly executed. As a result, the position of the suspended load 7 is sequentially detected, and the suspended load 7 can be tracked.

However, in step 208, the correlation value obtained in step 205 is compared with a predetermined threshold value to determine whether an error has occurred. If it is determined that there is no error, the updated coordinate position (X1 + VX1,
Y1 + VY1) is set again in the search window SW to perform pattern matching.

However, if it is determined that there is an error, the process of searching for the target mark 8 with the search window SW left at the current coordinate position (X1, Y1) is performed.
This is performed in the same manner as step 202. After searching the target mark 8, pattern matching in step 204 is performed.

Now, the brightness data of each pixel of the reference block RB is changed every time pattern matching is performed.
It is updated so that the brightness becomes according to the brightness of the pixel of the matched block B.

That is, when the process reaches step 206, the subroutine shown in FIG. 15 is executed.

Now, in each pixel of the reference block RB, as shown in FIG. 11, 0 from the upper left to the lower right.
They are numbered from 1 to 255.

Therefore, the brightness of the pixel is classified into three as follows.

A set of pixel numbers corresponding to a white portion a A set of pixel numbers corresponding to a black portion b A set of pixel numbers corresponding to an intermediate between white and black c In each of the above sets, for example, a is a pixel 0 1, 2, 5, 6, 20, to 134 b are classified into pixels 7, 9, 32, 34, 45, to 255 c, and pixels 15, 16, 52, 94, 158, and 204 are classified. Here, the average of the gray level values of the pixels corresponding to the sets a, b, and c in this reference block RB is obtained. That is, the average of the gray level values of the pixels of the set a in the current reference block RB AR The average of the gray level values of the pixels of the set b in the current reference block RB BR The average gray level value of the pixels of the set c in the current reference block RB CR Is calculated (steps 301, 302, 303).

Next, as a result of the pattern matching, the coordinate position (X1 + VX1, Y1 + VY1) of the matching block B (gaze window B) having the highest degree of correlation is moved by the moving amount V (VX1, VY1) obtained in step 204. (Step 304).

Now, suppose that the gaze window B is as shown in FIG. Here, similarly to steps 301 to 303, the average of the gray level values of the pixels corresponding to the sets a, b, and c in the gaze window B is obtained. That is, the average AT of the intensity values of the pixels of the set a in the watching window B, the average AT of the intensity values of the pixels of the set b of the watching window B, BT, and the average CT of the intensity values of the pixels of set c in the watching window B (( Steps 305, 306, 307).

The current gaze window B reflects the latest illuminance change. Therefore, to update the current reference block RB to match the latest illuminance,
All pixels of the current reference block RB (25
The following conversion is performed for 6 pieces).

For each pixel in set a, AT / AR
Double.

For each pixel in set b, BT / BR
Double.

For each pixel of set c, {(AT-B
T) / (AR−BR)} × (shading value of each pixel−BR)
+ BT Thus, the brightness data of the reference block RB is updated.

FIG. 13 is a graph for explaining the above conversion.

The horizontal axis represents the gray level value of the previous gaze window B, that is, the current reference block RB, and the vertical axis represents the gray level value of the current gaze window B, that is, the updated reference block RB. . The brightness AR of the white pixel is converted into AT, the brightness BR of the black pixel is converted into BT, and the intermediate portion between white and black is converted by being interpolated by the straight line N = αM + β (steps 308, 309,
310).

As described above, according to the second embodiment, every time the pattern matching is performed, the brightness corresponding to the brightness of the pixel of the matched block B (the captured image at that time) is obtained. Reference block RB
Will be updated. In this way, normalization (offset, which reflects the latest brightness information in the reference block RB,
Since (gain absorption) is always performed, even in an environment where the illuminance changes significantly, such as outdoors, correlation can be easily obtained, and the remarkable effect of dramatically improving the accuracy of pattern matching can be obtained. As a result, tracking can be performed stably.

Moreover, since the above-mentioned normalization process only needs to be performed for the local area (16 × 16 pixels) of the captured image, there is an advantage that the amount of calculation is extremely small and the process can be advanced at high speed.

Third Embodiment The third embodiment described below is the same as the second embodiment in that the reference block RB is updated, but the way of updating is different. It is a thing.
In the following, description will be made while omitting the description overlapping with that of the second embodiment.

FIG. 16 is a diagram for explaining the pattern matching performed in the third embodiment.

That is, the picked-up image 30 and the reference block RB which is the reference pattern image showing the target mark 8 are butted against each other, and the correlation therebetween is taken to perform pattern matching (template matching). Then, the captured image 30 when the correlation becomes the highest
It is determined that the suspended load 7 exists at the coordinate position of the block B, and the detected position of the suspended load 7 is output. The above points are the same as in the second embodiment.

The updating method is performed by taking out the matched block B from the picked-up image 31 and replacing the current reference block RB with the taken out matching block B each time the matching is performed. This point is different from the second embodiment.

Next, a processing procedure will be described with reference to FIGS.

As shown in FIG. 18, steps 401, 4
02, 404, 405, 407, 408, 409,
Steps 201, 202, 204, 205 and 2 in FIG.
The same processing as 07, 208, and 209 is executed.

At step 403, the initial coordinate position (X
The point that one reference block of (1, Y1) (this is referred to as RB0) is created is the same as in step 203 above, but a plurality of other reference blocks are created for correction of the coordinate position described later. I keep it.

That is, in FIG. 17, when RB is the reference block RB0, a total of 16 reference blocks such as reference blocks RB1, RB2, ... RB16 around coordinate positions P0, P1, P2 ,. Is set (step 403).

Then, the reference block RB0.
Is used to perform pattern matching in the same manner as in step 204 (step 404).

At step 406, the subroutine shown in FIG. 19 is executed.

That is, the movement amount V at the time of matching
From (VX1, VY1), the coordinate position (X1 + VX1, Y1 + VY1) of the matched block B is calculated (step 501). And the current reference block R
B0 is replaced with the matched block B itself, and this is used as a new reference block RB (step 502).

Now, simply refer to the reference block RB as
If only the matching blocks B are sequentially replaced, a position error smaller than one pixel is accumulated, so that a position error of one pixel or more occurs. Therefore, the following processing is performed in order to correct this position error.

That is, using the normalized correlation between the replaced reference block RB and the original reference block RB0 initially set,
The degree of correlation (0) is calculated (step 503).

Next, the replaced reference block RB and the original blocks RB1, RB2, ... Surrounding the originally set original block RB0.
Using the normalized correlation with RB16, the respective correlation degrees D (1), D (2), ... D (16) are obtained.

The normalized correlation can be calculated, for example, in (29) below.
Based on the formula (step 503).

[0121] (29) Next, the correlations D (0) to D (16) are compared, and the one having the highest correlation is searched (step 505).

If D (0) indicates a value with the highest correlation, it is determined that no position error has occurred, and the reference block RB replaced in step 502 is replaced with the updated reference block. It is determined that there is, and the process returns to step 407 of FIG. 18 (YES at step 506).

However, D (1) to D (1 other than D (0)
If any of 6) indicates a value with the highest correlation (No in step 506), it is determined that a position error has occurred, and the coordinate position of the original block with the highest correlation is determined. Set to the corrected position.

That is, for example, if D (1) is a value indicating the highest correlation, the deviation between the coordinate position of the corresponding original block RB1 and the coordinate position of the original block RB0 used for the actual pattern matching. (Dx, Dy) is calculated (step 507).

Then, the movement amount V acquired in step 501 is corrected by the above deviation as shown in the following equation (30).

VX1 = VX1−DX VY1 = VY1−DY (30) Then, as shown in FIG. 17, the reference block RB replaced in step 502 has coordinates displaced by the deviation (Dx, Dy). The block B existing at the position P1 is modified. That is, the original block RB
The block B corresponding to the coordinate position of 1 is modified. Thus, finally, the block B existing at the coordinate position P1
Is determined to be the reference block RB, and
Return to step 407 (step 508).

As described above, according to the third embodiment, every time pattern matching is performed, the reference block RB is replaced by the matching image at that time, and the reference block RB is updated. Therefore, similar to the second embodiment, the accuracy of pattern matching is dramatically improved even in an environment where the illuminance changes greatly, such as outdoors, and as a result, tracking can be stably performed. The effect is obtained.

Moreover, every time the pattern matching is performed, the positional error of the matching image is corrected by checking the correlation with the original reference block, so that it is possible to avoid the occurrence of the positional deviation.

Fourth Embodiment Next, a control device for controlling the zoom amount of the camera 9 shown in FIG. 1 will be described.

Now, as in the second and third embodiments,
Reference block RB showing the target mark 8
And the captured image including the target mark 8 are matched with each other to perform pattern matching, the size of the target mark 8 in the captured image, that is, the ratio between the size of the captured image and the size of the target mark 8 is extremely small. It becomes important. That is, the size of the target mark 8 in the captured image needs to be the same as the size of the target mark 8 of the reference block RB.

However, in the crane vehicle 1 having the structure in which the camera 9 is attached to the boom tip 2a and the target mark 8 suspended by the camera 9 via the rope 5 is imaged, the length of the rope is The change in Lp changes the size of the imaged target mark 8.

Therefore, as shown in FIG. 23, when the size of the target mark 8 in the captured image 32 matches the size of the target mark 8 of the reference block RB, good pattern matching can be performed. However, if the size of the target mark 8 in the captured image 32 does not match the size of the target mark 8 of the reference block RB, pattern matching cannot be performed well.

Therefore, in the third embodiment, the camera 9
By controlling the zoom amount of, the ratio of the size of the captured image 32 and the size of the target mark 8 in the image 32 is always set to a desired constant value, that is, the “matching” value in FIG. Try to do well.

FIG. 20 is a block diagram showing the configuration of the embodiment, in which the rope length sensor 23 outputs a signal indicating the current rope length Lp to the interlocking device 22. In the interlocking device 22, the processing shown in FIG. 21 is executed and the zoom command Sz is sent to the camera 9. The camera 9 changes the zoom amount according to the input zoom command Sz, and captures an image including the target mark 8. The image signal output from the camera 9 is output to the image processing device 13,
For example, the processing shown in the first to third embodiments is executed.

Now, the relationship between the rope length Lp and the zoom command Sz will be considered.

Generally, the relationship between the distance to the object and the zoom is obtained from the equation of the lens image formation of geometrical optics. For the sake of simplicity, considering the case of one lens, the formula for lens imaging is (1 / a) + (1 / b) = (1 / f) (31) where a: from lens to object B: Distance from lens to image f: Focal length of lens At this time, the image magnification R represented by the ratio of the size of the object to the image is R = Hb / Ha = b / a (32) from FIG. By substituting the above equation (31) into the above equation (32) and substituting it into the above equation (32), Hb = {Ha / (af)} × f (33) , A, the zoom command Sz may be given to the camera 9 so as to change the focal length f of the lens.

That is, when the focal length when the rope length Lp changes from Lp by ΔLp is f ', the above (3
From equation (3), in order to make Hb constant, the following (34)
It is necessary to satisfy the relationship of the formula.

{F / (Lp−f)} = {f ′ / (Lp + ΔLp−f ′)} (34) Therefore, the focal length f ′ is calculated from the equation (34) as f ′ = {(Lp + ΔLp) / If Lp} × f (35), Hb has a constant size, that is, the target mark 8 to be imaged by the camera 9 has a constant size.

Generally, the focal length f cannot be changed with one lens assumed here, but since the general TV camera 9 is composed of several groups of lenses, it can be used as a lens system. The focal length f can be changed.

Therefore, the relationship (equation (35)) between the variation ΔLp of the rope length Lp and the zoom amount Sz is obtained in advance by simulation or the like, and the obtained rope length L
The correspondence between the change amount ΔLp of p and the zoom amount Sz is stored in the interlocking device 22.

The interlocking device 22 executes the processing shown in FIG.

First, at the start of zoom control of the camera 9, the initial rope length Lp0 is detected and stored (step 601).

Then, the work is started in the mobile crane 1 and the rope length Lp changes, but the rope length detection value is successively input from the sensor 23 and the accurate rope length Lp is calculated (step 602). Then, the deviation between the current rope length Lp and the initial rope length Lp0 is obtained, and the rope length change length ΔLp is calculated (step 603).

Then, the zoom amount Sz corresponding to the rope length change length ΔLp is read from the initially stored correspondence between the change amount ΔLp of the rope length Lp and the zoom amount Sz.

In this case, the rope length change amount ΔLp is compared with a predetermined threshold value, and the zoom amount Sz is read only when the rope length change amount ΔLp is equal to or more than the threshold value. Should not be read. This is for avoiding a situation in which the zoom amount of the camera 9 is sensitively changed due to a slight change in the rope length and hunting or the like occurs.

It is also possible to prepare an arithmetic expression for calculating the zoom amount Sz from the change amount ΔLp of the rope length Lp based on the equation (35), and calculate the zoom amount in real time (step 604).

When the zoom amount Sz is obtained in this way, the zoom command Sz for setting this zoom amount is output to the camera 9.

As a result, even if the rope length Lp changes during the crane work, the zoom amount of the camera 9 is changed and the size of the target mark 8 picked up by the camera 9 is kept constant. Be drunk

Therefore, as shown in FIG. 23, the captured image 3
The size of the target mark 8 in 2 always matches the size of the target mark 8 of the reference block RB, so that the image processing apparatus 13 can always perform good pattern matching.

In the first to fourth embodiments described above, it is assumed that the present invention is applied to a mobile crane, but any working machine that suspends a load from the tip of the boom can be used. Applicable.

Further, in the embodiment, the target mark is imaged and the pattern matching with the pattern of the target mark is performed instead of the image of the suspended load. It is also possible to take an image of the object, the suspended load itself, the hook, and perform pattern matching with these patterns.

[0152]

As described above, according to the first invention corresponding to the above-mentioned first embodiment, the position data of the suspended load obtained by the image processing is the minimum value that only the position of the control start is detected. Since the stop control of the shake of the load is performed without feeding back the position of the suspended load, the shake stop control of the suspended load is accurately performed while avoiding the inconvenience caused by the feedback control.

Further, according to the second invention corresponding to the second and third embodiments, even in an environment such as outdoors, where the brightness constantly changes, the picked-up image of the suspended load is displayed. Since the lightness of the pixels of the pattern image is updated so that the lightness of the pixels and the lightness of the pixels of the pattern image of the hanging load are sequentially the same, the pattern matching is performed accurately, and the position of the hanging load is thus adjusted. Can be tracked accurately.

Further, according to the third invention corresponding to the above-mentioned fourth embodiment, even if the rope length changes due to crane work or the like, the ratio of the size of the suspended load to the image picked up by the image pickup means is always Since the desired constant value is set, processing such as pattern matching can be performed efficiently and without increasing the storage capacity.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a mobile crane to which a picked-up image processing apparatus for a suspended load according to the present invention is applied.

FIG. 2 is a block diagram showing a configuration of a first embodiment device.

FIG. 3 is a graph showing a change over time of a swing of a suspended load.

FIG. 4 is a physical model diagram showing a pendulum motion of a suspended load.

FIG. 5 is a diagram showing a speed change pattern of a boom of a mobile crane.

FIG. 6 is a flowchart showing a procedure of processing performed by the device according to the first embodiment.

FIG. 7 is a graph showing a simulation result of the first embodiment.

FIG. 8 is a block diagram showing a configuration of a second embodiment device.

FIG. 9 is a diagram used for explaining the concept of the second embodiment.

FIG. 10 is a diagram used for explaining pattern matching applied to the embodiment.

FIG. 11 is a diagram showing the brightness of pixels of a reference block.

FIG. 12 is a diagram showing the brightness of pixels of a matched block.

FIG. 13 is a graph illustrating how the brightness data is updated.

FIG. 14 is a flowchart illustrating a procedure of processing performed by the device according to the second embodiment.

FIG. 15 is a flowchart showing a procedure of a subroutine process of FIG.

FIG. 16 is a diagram used for explaining the concept of the third embodiment.

FIG. 17 is a diagram showing coordinate points existing around the updated reference block.

FIG. 18 is a flowchart showing a procedure of processing performed by the device of the third embodiment.

FIG. 19 is a flowchart showing a procedure of a subroutine process of FIG.

FIG. 20 is a diagram showing the configuration of a fourth embodiment device.

21 is a flowchart showing a procedure of processing performed by the interlocking device shown in FIG. 20.

FIG. 22 is a diagram for explaining the principle applied to the fourth embodiment.

FIG. 23 is a diagram showing a comparison between a pattern size of a reference block and a pattern size of a captured image.

[Explanation of symbols]

 2 Boom 5 Rope 7 Suspended load 8 Target mark 9 Camera 13 Image processing device

Claims (5)

[Claims] 1. An image of a suspended load suspended at the tip of a boom is captured, and the captured image is processed,
A picked-up image processing device for a suspended load, which detects a position of a suspended load on an image, and based on the detected position of the suspended load on the image, performs shake stop control for stopping the shake of the suspended load, Obtaining a motion equation describing the relationship between the swing angle and swing angular velocity of the load and the moving acceleration of the boom tip,
Based on the equation of motion, the boom tip from the time when the swing angle of the suspended load reaches a predetermined swing angle at which the swing stop control should be started to the time when the swing angle and the swing angular velocity of the suspended load simultaneously become zero. By moving speed change pattern generation means for generating a moving speed change pattern showing the state of change of the moving speed of the, and by tracking the position of the suspended load on the image, the swing angle of the suspended load becomes the predetermined swing angle. And a moving speed change pattern generated by the moving speed change pattern generating means from the time when the control start time detecting means detects that the predetermined swing angle has been reached. And a moving speed control means for controlling a moving speed of the boom tip according to the above. 2. The position of the suspended load on the captured image is determined by capturing an image of the suspended load suspended at the tip of the boom and sequentially performing pattern matching between the captured image and a pattern image showing the suspended load. In the picked-up image processing device for a suspended load that is sequentially detected, the brightness of the pixels of the pattern image is the same as the brightness of the pixels of the picked-up image when the picked-up image and the pattern image match. An image processing apparatus for picked-up image of a suspended load, wherein the brightness data of pixels of the pattern image is sequentially updated every time matching is performed. 3. The position of the suspended load on the captured image is determined by capturing an image of the suspended load suspended on the tip of the boom and performing pattern matching between the captured image and a pattern image indicating the suspended load. In the picked-up image processing device for the hanging load that is sequentially detected, every time the picked-up image and the pattern image match, a matching image of a portion corresponding to the pattern image is taken out from the picked-up image, A picked-up image processing apparatus for a suspended load, wherein the pattern image is sequentially updated by replacing the pattern image with the extracted matching image. 4. The position of the suspended load on the captured image is determined by capturing an image of the suspended load suspended at the tip of the boom and performing pattern matching between the captured image and a pattern image showing the suspended load. In the picked-up image processing apparatus for the suspended load that is sequentially detected, a center pattern image indicating the suspended load is extracted in advance from the captured image of the suspended load, and the center existing near the coordinate position of the center pattern image is extracted. A peripheral pattern image of the same size as the pattern image is extracted in advance, pattern matching is performed between the captured images that are sequentially captured and the central pattern image, and each time the captured image and the central pattern image match, the A matching image of a portion corresponding to the central pattern image is extracted from the captured image, and the extracted matching image is , Taking a normalized correlation with the central pattern image and the surrounding pattern image,
Select the center pattern image or the surrounding pattern image with the highest normalized correlation, when the center pattern image is selected, the current pattern image, by replacing the extracted matching image, While sequentially updating, when the surrounding pattern image is selected, the current pattern image is a surrounding image of the same size as the matching image existing near the coordinate position of the extracted matching image, A hanging image pickup image processing apparatus, wherein the pattern image is sequentially updated by replacing it with a surrounding image at a coordinate position corresponding to the coordinate position of the selected surrounding pattern image. 5. A predetermined process is performed by capturing an image of a suspended load suspended at the tip of a boom with an image capturing means, and keeping a ratio between the size of the captured image and the captured suspended load constant. In the picked-up image processing apparatus for hanging load, a rope length detecting means for detecting a rope length of the hanging load, and a rope length of the hanging load detected by the rope length detecting means are imaged by the imaging means. Image size,
A picked-up image processing apparatus for a suspended load, comprising: a control unit that changes a zoom amount of the imaging unit so that a ratio with the size of the captured suspended load becomes a desired constant value.