JPH01320921A - Guiding device of working machine for fruit - Google Patents

Abstract

(57) [Summary] This bulletin contains application data before electronic filing, so abstract data is not recorded.

Description

DETAILED DESCRIPTION OF THE INVENTION [Industrial Application Field] The present invention provides an imaging device for capturing an image of a fruit tree, which is provided in a capturing section for capturing fruit, and based on image information obtained by imaging the fruit tree by the imaging device, a region extraction means for extracting each of a fruit region corresponding to a leaf, a leaf region corresponding to a leaf, and a branch region corresponding to a branch; a feature point extraction means for extracting geometric feature points in the fruit region; The present invention relates to a guiding device for a fruit tree working machine, which is provided with a guiding means for guiding the center of the trapping section toward a direction in which the geometrical characteristic point is located based on information from a characteristic point extracting means.

[Conventional technology]

The above-mentioned guidance device of this type of fruit tree working machine is designed to guide the fruit-trapping machine toward the direction where the geometric feature points of the fruit area are located, not corresponding to the fruit extracted based on the image information of the fruit tree. When guiding the capture unit, in addition to the fruit area, it is also possible to extract leaf areas corresponding to leaves and branch areas corresponding to branches, and check whether these leaf areas and branch areas are in contact with the fruit area on the image. Based on this, it is possible to determine whether or not the leaf or tip is within the guide path of the trapping unit to the target fruit (as proposed in the patent application filed in 1983 by the applicant earlier).
3-32332).

By the way, the captured image information is two-dimensional information without distance information, and it is not possible to determine the size relationship with respect to the capturing part only from the information on the image of the leaf area and branch area extracted from the captured image, so it is difficult to capture the fruit. It was not possible to accurately determine whether this was possible or not.

To explain, even if the actual sizes of leaves and branches are the same, the sizes of the corresponding leaf regions and branch regions on the image change depending on the imaging distance. Therefore, when the imaging distance is long, even if the leaf or branch area overlaps the fruit area, it is actually possible to capture the fruit as long as the capturing unit is located outside the area to capture the fruit. It is. On the other hand, even if the leaf area or branch area slightly overlaps the fruit area, if the imaging distance is short, the leaf area or branch area is located within the area where the fruit is captured. In reality, it may be in a state where it cannot be captured. In other words, without distance information, it is difficult to accurately determine whether or not the fruit can be captured using only information about each area on the image.

For this reason, conventionally, even when a leaf region or a branch region overlaps with a fruit region, the capturing section is guided toward the fruit, and if the fruit cannot be captured as desired, it is determined that the fruit cannot be harvested. (For example, see Japanese Patent Application Laid-Open No. 63-102611.) [Problems to be Solved by the Invention] The present invention has been made in view of the above circumstances, and its purpose is to capture fruit based on captured image information. The purpose is to avoid wasteful guidance of a trapping unit as much as possible by making it possible to accurately determine whether or not it is possible.

[Means to solve the problem]

In the guidance device for a fruit tree working machine according to the present invention, an imaging means for imaging a fruit tree is provided in a fruit capturing section, and based on image information obtained by imaging the fruit tree by the imaging means, a fruit area corresponding to the fruit, A region extraction means for extracting each of a leaf region corresponding to a leaf and a branch region corresponding to a branch, a feature point extraction means for extracting geometric feature points in the fruit region, and information on the feature point extraction means guiding means for guiding the center of the trapping section toward the direction in which the geometrical feature point is located based on the guiding means, the characteristic structure of which is that the trapping section is guided toward the fruit to be guided. The catching section is provided with a distance determining means for determining whether or not the fruit has approached within a set distance, and based on the information of the distance determining means, the catching section is moved to the front side of the target fruit by the set distance. Information on each region and the geometric feature point extracted from image information captured in a state where the
The leaf area or The present invention is characterized in that, in the case where the branch area is present, a catchability determining means is provided which determines that the fruit cannot be caught.

[Function] Since the imaging means is provided in the capturing section, based on the information of the distance determining means provided in the capturing section, the capturing section is imaged in a state where the capturing section is located a set distance in front of the fruit to be guided. By doing so, the size of the capture portion on the captured image can always be determined as an area of the same size.

Therefore, by capturing an image with the capture unit positioned a set distance in front of the target fruit, the leaf area or branch area extracted from the captured image information will be within the area corresponding to the capture unit on the same captured image. In some cases, it is determined that the fruit cannot be captured (see FIG. 3(c)).

〔Effect of the invention〕

Therefore, it is possible to accurately determine whether or not the fruit to be guided can be captured, so that it is possible to avoid wastefully guiding the capturing section to fruits that cannot be captured. In addition, it is possible to prevent the catching part from colliding with a branch that gets in the way of catching and damaging the catching part or the fruit tree. As a result, it has become possible to accurately guide the trapping unit toward only the target fruit that can be captured.

〔Example〕

Embodiments of the present invention will be described below based on the drawings.

As shown in FIG. 4, a boom (2) that can rise and fall and swing left and right is pivotally attached to a traveling body (V) equipped with traveling wheels (1), and at the tip of the boom (2), Auxiliary boom (
3) is pivoted so as to be horizontally swingable, and a multi-jointed telescopic arm (4) is attached to the tip of the auxiliary boom (3) so that the direction can be changed in both the vertical and horizontal directions. , a working machine for fruit trees is constructed.

At the tip of the telescoping arm (4), there is a fruit to be harvested (
F) Power (5) as a trapping unit that adsorbs
A working hand (6) for fruit harvesting is attached. A color image sensor (S) as an imaging means is installed inside the vacuum pad (5).
l), the direction of the imaging field of view is the vacuum pad (5).
), and it is detected that the vacuum pad (5) has approached within a set distance (set to 30 cm) of the fruit to be guided, that is, to be harvested (about January). Proximity sensors (S2) as distance determination means using infrared light are respectively attached so that the direction of their sensing range matches the direction of the vacuum pad (5).

To explain the structure of the working hand (6), as shown in FIG. A trap case (8) is externally supported on the trachea (7) so that it can move forward and backward, and the handle of the fruit (F) taken into the trap case (8) is clamped in the vertical direction. It is equipped with a handle cutting device (9) that cuts the handle in the engulfed state.

The ventilation pipe (7) is connected via a bellows-type hose (10) (see FIG. 4) to a suction pump (not shown) mounted on the body (V).

The trapping part case (8) includes an upper case (8b) fixed to a support member (8a) that is fitted onto the ventilation pipe (7).
and a lower case (8C) which is pivotably attached to the support member (8a) around a vertical axis. That is, by swinging the lower case (8C) in the Chinese direction with respect to the ventilation pipe (7),
The fruit (F) whose handle has been cut, which is held within the trapping part case (8), can be discharged to the outside.

Further, the handle cutting device (9) includes a clipper-shaped blade (9a) for cutting the handle of the fruit (F), and the blade (9a).
and a handle support member (9b) that presses and supports the handle of the fruit (F) from below.

However, although not shown, the working hand (6) includes an actuator for moving the catching part case (8) in and out, an actuator for driving the handle cutting device (9), etc. various actuators for operating the harvest, a sensor for detecting that the fruit (F) has been adsorbed on the vacuum pad (5), a sensor for detecting the exit/retraction position of the trapping part case (8), and , various sensors such as a sensor for detecting the driving state of the handle cutting device (9) will be provided.

In the figure, (11) is an illumination device for illuminating the front side of the working hand (5) with a set light amount in synchronization with the imaging process of the image sensor (Sl), and the light emitting part (11) A temperature sensor (S3) is provided so that when the temperature of Ila) becomes higher than the set temperature, the illumination operation can be temporarily stopped until it cools down to the set temperature or lower. In other words, although not described in detail, since the illumination device (11) repeatedly emits light in synchronization with the imaging processing of the image sensor (Sl), if the light emission interval is short,
Because there is a risk that the temperature may rise abnormally and reduce durability, the system is designed to automatically temporarily pause the imaging process for safety purposes when the temperature exceeds the set temperature.

Then, with the working hand (6) facing the direction in which the fruit to be harvested is located, the image sensor (S) performs imaging processing and images while illuminating with the illumination device (11). Based on the image information, the working hand (
6), the approach direction, that is, the guiding direction of the working hand (6) is determined.

After determining the approach direction, use the work hand (6
) is actuated to move toward the determined approach direction, the actuator (not shown) of the telescoping arm (4) is actuated, and the proximity sensor (S2) is
N operation, the work hand (6) is re-imaged while the movement is stopped, and, as will be described in detail later, it is determined whether or not capture is possible based on the re-imaged image information. .

If it is determined that the fruit (F) to be harvested can be captured, the work hand (6) resumes movement, and the vacuum pad (5) is operated to suck the fruit (F) to be harvested;
After the fruit (F) is taken into the trapping part case (8), the stalk is cut and harvested.

However, if there are multiple fruits within the imaging field of the image sensor (Sl), the image sensor (Sl) determines which fruit should be approached first, as described below.
), the approach order is automatically determined based on the brightness information of the captured images.

Next, a control configuration for guiding the working hand (5) by determining the direction in which the fruit (F) is located, that is, the approach direction, based on image information captured by the image sensor (Sυ) will be described.

As shown in FIG. 1, the image sensor (Sl) is
It is configured to simultaneously output three primary color information (R), (G), (B) and luminance information (Y) as brightness information.

Then, extracting a fruit region (C1) corresponding to the fruit (F) and a leaf region (C2) corresponding to the leaf (H) using the three primary color information (R), (G), and (B), It is configured to extract a branch region (C4) corresponding to a branch (IE) by using information on the fruit region (C1) and leaf region (C2) and the luminance information (Y).

To explain, as shown in Figure 3 (c) and (d),
Since fruit (F) is generally a warm color, red information (R) includes color information corresponding to fruit (F) among the three primary color information (R), (G), and (B). The fruit area (C,
). Similarly, from green information (G) including color information corresponding to leaf (H) among the three primary color information (R), (G), and (B), red information (R) as other color information is obtained. By subtracting and binarizing, the leaf region (C2) corresponding to the leaf (H) is extracted.

However, although the branch (E) is generally brown in color, it may have a greenish color due to the adhesion of moss, etc., so the branch (E) may have a brownish color, but it may have a greenish color, such as the fruit area (C,) or the leaf area (C2). Then, using the three primary color information (R), (G), and (B), the wave (E
) cannot directly extract the branch region (C4) corresponding to the wave (C4), but the wave (Snake) is not the fruit region (C1) nor the leaf region (C2) in the region (C3). Not a background with shadows etc. that is darker than the brightness of E),
In addition, since the area is not a background such as the sky which is brighter than the brightness of the wave (E), the brightness information (Y) and the area (which is neither the fruit area (C1) nor the leaf area (C2)) C3).

Then, a fruit region extraction circuit (100) as fruit region extraction means extracts a fruit region (C1) corresponding to a fruit (F) from the red information (R) and the green information (G).
, from the green information (G) and the red information (R), leaf (
A leaf region extraction circuit (101) as a leaf region extraction means for extracting a leaf region (C2) corresponding to l(), the luminance information (Y), the fruit region (C1) and the leaf region (
Based on the information of C2), the branch area (
A branch area extraction circuit (102) as a branch area extraction means for extracting the fruit area (C1), a first image memory (12a) that stores image information corresponding to the fruit area (C8), and a first image memory (12a) that stores image information corresponding to the fruit area (C8);
A first image memory J that stores image information corresponding to C2)
(12b), a third image memory (12c) that stores image information corresponding to the branch area (C1), and a third image memory (12c) that stores image information corresponding to the branch area (C1);
) into a quantized density value in a setting stage corresponding to each pixel, and an A/D converter (13) that stores the density value converted by the A/D converter (13). 4 image memory (12d), and each of the image memories (12a),
(12b).

Based on the image information stored in each of (12c) and (12d), the direction in which the fruit (F) is located is determined, and it is also determined whether or not the fruit (F) to be harvested can be captured. A control device (14) using a microcomputer that constitutes various means for guiding the hand (6)
Each of these is provided.

In FIG. 1, (15) is an electronic shutter using a liquid crystal, which exposes the image sensor (S1) only for a set time in synchronization with the imaging process of the image sensor (S1). It has become. In addition, (16) is the boom (
2), an actuator for operating each of the auxiliary boom (3), the arm (4), and the working hand (6), which is determined using the control device (14); Based on the position information of the fruit (F), the working hand (6) is operated to guide toward the fruit (F).

That is, the feature point extraction means (103) extracts the geometric feature points (P) in the fruit region (C3) using the control device (14);
) guiding means (1) for guiding the center of the vacuum pad (6) as a trapping section toward the direction in which the geometric feature point (P) of the fruit fishing season (C1) is located;
04), and each of the above regions (C,), (C2), C
4> and information on the geometrical feature point (P), and imaging of the vacuum pad (5) in a state where the vacuum pad (5) serving as a capturing section is located at the set distance in front of the fruit. Based on the size information on the image, if the leaf area (C2) or the branch area (C1) is in the area corresponding to the vacuum pad (5), the fruit (F) is removed. Each of the capture impossibility determining means (105) that determines that capture is not possible is configured.

Next, the configuration of each part will be explained in detail.

The fruit area extraction circuit (100) includes a first subtracter that subtracts green information (G) that does not include the color component of the fruit (F) from red information (1?) that includes the color component of the fruit (F). (
17) and the signal whose value of the output signal of the first subtractor (17) is larger than a first set threshold (THDI) set in advance corresponding to the color of the fruit (F) is binarized. and a first comparator (18) for extracting the fruit area (c's) (see FIG. 3(d)).

The leaf region extraction circuit (101), like the fruit region extraction circuit (100), extracts the color component of the leaf (II) from the green color information (G) that includes the color component of the leaf (H). A second subtracter (19) that subtracts the red color information (R) and the value of the output signal of the second subtractor (19)
The second setting threshold value (TH
D2), the signal that is larger than the leaf area (C2) is binarized and
(See FIG. 3(d)).

The branch region extraction circuit (102) extracts the luminance information (Y
A third comparator ( 21) and the value of the luminance information (Y) is the wave (E)
the fruit area extraction circuit ( The fruit area (100) output from the fruit area (
C1) and the leaf region (C2) outputted from the leaf region extracting circuit (101), that is, a region that is not the fruit region (C1) and the leaf region (C
2) Compute the signal corresponding to the region (C3) that is not
It consists of an OR gate (23) and an AND gate (24) that calculates the logical product of the output signal of the NOR gate (23) and the output signals of both the comparators (21) and (22).

In other words, the branch area (C4) (see FIG. 3(d)) is
The fruit region (Cυ) and the leaf region (C2
) is brighter than the third setting threshold value (TM01) for the shadow, etc., and the fourth setting darkness value (T) for the sky, etc.! It is extracted as a region darker than D4) and output from the AND game (24).

By the way, even if an image is taken with the work hand (6) positioned at the approach start position, since there is no distance information between the image sensor (Sl) and the fruit (F), each area (Cυ , (C2), (C4) and the size of the range where the vacuum pad (5) sucks the fruit (F),
), (12b), (12c), and (12d), it is difficult to determine whether or not the vacuum pad (5) can suck the fruit (F). It is not possible.

Therefore, using the information of the proximity sensor (S2), when the working hand (6) reaches a set distance at which the proximity sensor (S2) detects the approach to the fruit (F), -
The images are then stopped once and re-imaging processing is performed, and each of the regions (C1), (C2),
(C4) is re-extracted to determine whether or not it can be captured.

That is, the working hand (6) is connected to the proximity sensor (
When S2) is located at a set distance in front of the fruit (F) that detects the approach to the fruit (F), the target fruit (F)
), the vacuum pad (5) becomes a constant value as shown in FIG. 3(C).
Each of the first capture area (A, ) that adsorbs F) and the second capture area (A2) corresponding to the outer shape of the capture unit case (8) is determined as an area of a constant size on the captured image. be able to.

That is, the working hand (6) is connected to the proximity sensor (S).
2) on the captured image where the leaf area (C2) or the branch area (C4) is located at a set distance in front of the fruit (F) that detects the approach to the fruit (F), the first capture area (A1) and the branch area (C4)
is located inside the second capture area (A2), it can be determined that capture is impossible. Note that even if the leaf region (C2) is located inside the second trapping region (A2), the leaf region (C2) or the branch region (C1) is located inside the first trapping region (Aυ). If it is not on the inside, there is no risk that the vacuum pad (5) will adsorb the leaf (It) (and the leaf (H) can be captured because it will not get in the way of guidance.

Next, the operation of the control device (14) will be explained based on the flowchart shown in FIG.

First, the work hand (6) is set at a preset approach start position, and image processing is performed, and the image information (see FIG. 3 (A)) is used to image the fruit area corresponding to the fruit (F). (C1) and the value of the brightness information (Y), the geometric feature point ( P) (see Figure 3 (b)).

To explain further, since the fruit (F) is a spherical body, it is known that there is a single point that appears brightest on its surface. Therefore, the fruit (F
), i.e., the geometric feature point (
P) can be used as information representative of the direction in which the fruit (F) is located.

However, as shown in FIG. 3 (B), if a plurality of fruit regions (C1) exist in the image taken at the approach start position, the fruit regions (C1) of these plural fruit regions (
Based on the size of the area where the brightness in each of C1) is larger than the set value, extract the area with the largest size, and move the work hand (6) toward the largest set area. ), the arm (4) is extended by adjusting the direction of the working hand (6).

To explain, the brightness of the image information is inversely proportional to the distance, and the closer the image information is, the brighter it becomes. The farther away the object is, the larger it becomes. In other words, the brightness information (Y) is used to approach the fruit (F) that is located closer to the user and is easier to harvest.

After the arm (4) starts to extend, the work hand remains in position until the proximity sensor (S2) is turned on and detects that the fruit (F) to be harvested has approached within the set distance. The arm (4) is extended so that the arm (6) goes straight toward the fruit (F) to be harvested.

As the proximity sensor (S2) turns ON and detects that the fruit to be harvested has approached within the set distance, -
The first area (Aυ) corresponding to the vacuum pad (5) and the second area corresponding to the trapping part case (8) are temporarily stopped and re-imaged, and based on the imaged information, the first area (Aυ) corresponding to the vacuum pad (5) and the second area corresponding to the trapping part case (8) are (A2) Inside each of the leaf regions (
Based on whether or not C2) and the branch area (C4) are located, it is determined whether or not capture is possible, and if capture is not possible, the arm (4) is retracted and returned to the approach starting position. .

In other words, the process of determining whether capture is possible or not based on the image information re-captured with the vacuum pad (5) positioned a set distance in front of the fruit (F) to be harvested is the process of determining whether capture is possible or not. This corresponds to the means (105).

Note that the geometric feature point (P) is re-extracted based on the image information obtained by this re-imaging process, and the center of the vacuum pad (5) is aligned with the re-extracted geometric feature point (P). The guiding direction is corrected so that the robot moves in the direction in which point P) is located.

That is, a process of extracting a highlight point with maximum brightness in the fruit region (C1), which is extracted based on image information captured at the approach start position, as the geometric feature point (P); and in the fruit area (C1) extracted based on image information re-imaged with the vacuum pad (5) positioned a set distance in front of the fruit (F) to be harvested. Each of the processes for extracting the highlight point with the maximum brightness as the geometric feature point (P) corresponds to the feature point extraction means (103), and
The process of extending the arm (4) so as to guide the center of 5) toward the direction in which the geometric feature point (P) is located corresponds to the guiding means (104).

However, even if the arm (4) is extended until it reaches a preset distance, the proximity sensor (S2) will not turn on.
If it does not operate, it is determined that harvesting is not possible, and the arm (4) is retracted and returned to the approach starting position.

If it is possible to capture it, the working hand (6) will resume moving at a low speed, and the suction operation of the vacuum pad (5) will be started to perform a harvesting operation.

After starting the harvesting operation, it is determined whether the harvesting has failed based on the detection information of various sensors attached to the working hand (6), and if the harvesting has failed, the harvesting is not possible. It will be determined that there is one and will be returned to the approach starting position.

If the fruit (F) can be harvested properly, it is returned to the approach starting position and the harvested fruit is discharged.

After discharging the harvested fruits, it is determined whether or not the work is completed based on the number of fruit areas (C3) extracted from the image information first captured at the approach start position, and if the work is not completed, is the working hand (6)
This means that each process is repeated after the process of setting the camera at the approach start position and capturing an image.

[Another example]

In the above embodiment, the positional information of the highlight point with the maximum brightness in the fruit region (C1) is used as the geometric feature point (P), but the fruit region (C1 ) The center of gravity of a set area whose brightness is equal to or higher than a set value or the positional information of the center of gravity of the fruit area (C1) itself can also be used.

Further, in the above embodiment, the capture failure determining means (105) is
A first area (AI) corresponding to the vacuum pad (5)
) in which the leaf region (C2) or the branch region (C4) is located, and the second region corresponding to the trapping part case (8)
In each of the cases where the branch area (C4) is in the area (A2), it is determined that the branch area (C4) cannot be captured.
C2) or the branch area (C1) is present, it may be determined that capture is impossible.

Further, in the above embodiment, the present invention is applied to a fruit harvesting machine, but it can also be applied to various types of machines other than a harvester, and the present invention can be applied to a fruit area (Cυ, a leaf area (Cυ), a leaf area (C), etc.
2), and each area extraction means (10[1) for extracting each of the branch areas (C3).

Specific configurations of (101) and (102), specific configuration of the trapping section, specific configuration of the guiding means (104) for guiding the trapping section toward the fruit (F), and in carrying out the present invention. The specific configuration of each necessary part can be changed in various ways.

Incidentally, although reference numerals are written in the claims section for convenient comparison with the drawings, the present invention is not limited to the structure shown in the accompanying drawings.

[Brief explanation of the drawing]

The drawings show an embodiment of the guidance device for a fruit tree working machine according to the present invention, in which Fig. 1 is a block diagram of the control configuration, Fig. 2 is a flowchart of control operation, and Figs. 3 (a) to (d) are images. FIG. 4 is a schematic side view of the working machine, and FIG. 5 is a cross-sectional view showing the schematic configuration of the catching section. (Sl)...imaging means, (S2)...
Distance determination means, (F)...fruit, (H)...
...leaf, (ε) ...branch, (C1) ...
・Fruit area, (C2)...Leaf area, (C1)・
...Branch region, (P) ...Geometric feature point, (5) ...Capturing part, (100), (101
), (102)...area extraction means, (103
)...Feature point extraction means, (104)...
- Guidance means, (105)... Capturing impossibility determination means.

Claims (1)

[Claims] An imaging means (S_1) for taking an image of a fruit tree is provided in the fruit capturing part (5), and based on image information obtained by imaging the fruit tree by the imaging means (S_1), a fruit area (F) corresponding to the fruit (F) is determined. C_1), leaf area (
C_2) and a branch area (C_4) corresponding to the branch (E)
area extraction means (100) and (101) for extracting each of the
, (102), and feature point extraction means (103) for extracting geometric feature points (P) in the fruit area (C_1).
and guiding means (104) for guiding the center of the capturing section (5) toward the direction in which the geometrical feature point (P) is located based on the information of the feature point extraction means (103). In this guidance device for a fruit tree working machine, a distance determining means (S_2) for determining whether or not the catching part (5) has approached within a set distance from the fruit to be guided is provided. 5), and is extracted from image information captured in a state in which the capturing unit (5) is positioned at the set distance in front of the target fruit based on the information of the distance determining means (S_2). Each area (C_1), (
C_2), (C_4) and information on the geometric feature point (P), and imaging of the capturing unit (5) in a state where the capturing unit (5) is located at the set distance in front of the fruit. Based on the size information on the image, the capturing unit (
If the leaf area (C_2) or the branch area (C_4) is in the area corresponding to 5), a capture failure determining means (105) is provided for determining that the fruit (F) cannot be captured. Guidance device for fruit tree working machines.