JPH0926547A - Endoscopic image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately and easily estimate the absolute distance to an object, based on images obtained by picking up the same object from different positions. SOLUTION: The plural images obtained by picking up the same object from the different positions while moving an image pickup means are fetched and temporarily stored in a memory 61. Based on the plural images, the relative shape of the object and the distance to a proper position on the object from the image pickup means are estimated and the absolute shape of the object is estimated from the obtained relative shape of the object and the obtained distance to the object by a relative shape estimating means 62, a distance estimating means 63, and an absolute shape estimating means 64. The image data of the obtained shape of the object is temporarily stored in the memory 65 and inputted in a monitor 5 so as to display the three-dimensional shape image of the object.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an endoscopic image processing apparatus for estimating a distance from an image pickup means to a target object by using images of the same target object taken from different positions.

[0002]

2. Description of the Related Art The problem of estimating the shape of an object from a picked-up image when the relative arrangement of the image pickup means arranged so as to overlap the field of view is known, the so-called stereo image shape is used. Various methods have been proposed as estimation problems.

Furthermore, in recent years, several proposals have been made for extracting three-dimensional structures from motion. This is a method for trying to estimate the relative movement of the image pickup means from a plurality of images.

Each of these methods uses, as the original information, that the position of the same point on the object is associated with each other between a plurality of images.

Various methods have been proposed for detecting the position of the same point on an object on each image. In the case of artificial objects, since the corners and contour components are often clear, the method of structure extraction such as line segment extraction is effective, but it is difficult to apply to general natural images. The density gradient method, which is often used for time-series images, gives very good results when the apparent quality of the object between images is very small and the image quality is good, but the constraints of the imaging conditions are large. .

Therefore, in general, the point of interest on the image and its surrounding area are extracted as a reference area, and a correlation operation is performed between the area extracted from the image of the object to be searched to find the position giving the maximum correlation value. Corresponding point method (block matching)
Is often used. This block matching method gives a relatively stable result if the texture on the image is clear, but may detect an incorrect corresponding point if the texture is unclear.

As a further essential drawback, the detection result is unreliable when the object is a solid and the block includes a boundary with the background, for example. Further, it is also reliable when the deformation of the image between the images of the plurality of image pickup devices is large or the difference in the image size is large, such as when the object surface is tilted with respect to the image pickup device or there is a difference in distance. Can not.

The problems associated with the estimation of the three-dimensional shape include the problem of occlusion and the matching of estimation results. In particular, in stereo estimation, there is a part that is hidden behind the object and is not imaged, or a part that is imaged by only one of the image pickup means, and handling of these regions becomes a problem.

A large number of image pickup means inevitably results in a large area to be imaged and a small hidden area. However, especially when the position of the image pickup means is not known or the estimation is not accurate, these areas are hidden. Matching between images is not easy.

As described above, most of the methods that have been proposed in the past do not use an image of an artificial object as an object,
Or, by making one of the assumptions for these problems that naturally occur when a natural image is the object,
It is considered under conditions where the influence is excluded or mitigated, and it cannot be said that it has practically sufficient ability.

For example, there is an image obtained from a biomedical endoscope as a type of image that is difficult to apply the method proposed hitherto and is of great practical value when the function is realized.

By inserting an elongated insertion portion into a body cavity, an endoscope can be used for observing an affected area in the body cavity without the need for incision and for medical treatment using a treatment tool if necessary. However, since the size of the tip must be minimized in terms of function, only a member necessary for the doctor to perform observation or treatment can be incorporated.

Several proposals have already been made for grasping the shape of a target object endoscopically. For example, a method of projecting pattern light or the like onto an object to be observed (Japanese Patent Laid-Open No. Sho 6-96
No. 3-240831) and a system having a compound eye at the tip (Japanese Patent Laid-Open No. 63-244011), both of which require a special structure at the tip or the light source of the endoscope, Generally, it is difficult to use because it becomes large and complicated.

Japanese Patent Application Laid-Open No. 63-24671 by the same applicant
No. 6 discloses a method of estimating the shape of an object from a plurality of images obtained by moving the tip of the endoscope by hand operation, and a measuring mechanism for measuring the amount of movement of the tip associated with the operation. According to this method, an absolute object shape can be estimated without impairing the function of the current endoscope.

Further, Japanese Patent Laid-Open No. 6-728 by the same applicant
According to Japanese Patent Laid-Open No. 9, a reference object having a known size is placed near a target object, and a relative shape of the target object and the reference object is obtained from a plurality of images obtained by moving the tip of the endoscope by hand operation. And a method of estimating the size of the target object from the size of a reference object placed near the target object.

[0016]

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention
In the configuration of Japanese Laid-Open Patent Publication No. 246716, the relative shape of the target object is estimated from a plurality of images obtained in time series, and the movement amount of the image pickup means at the tip of the endoscope accompanying this operation is measured and the measured value is obtained. It is possible to estimate the absolute shape from

However, it is difficult to say that the accuracy that the measuring mechanism can provide is sufficient, if not sufficient, considering the resolution of the image pickup means.

Further, in the configuration of Japanese Patent Laid-Open No. 6-7289, when the target object is time-sequentially imaged, the reference object having a known shape size is simultaneously imaged together with the target object. It seeks an absolute shape.

However, there is a problem that placing a reference object near a target object by using a living body endoscope and capturing an image at the same time imposes a heavy burden on the operator of the endoscope.

The present invention has been made in view of these circumstances, and based on images taken from different positions with respect to the same target object, the absolute distance from the image pickup means to the target object can be accurately measured. An object is to provide an endoscopic image processing device that can be easily estimated.

[0021]

An endoscope image processing apparatus according to the present invention includes an endoscope having an image pickup means for picking up images of the same target object from different positions, and a plurality of images picked up by the image pickup means. Relative shape estimation means for estimating the relative shape of the target object from the image, distance estimation means for estimating the distance from the imaging means to an appropriate position on the target object, and estimation by the relative shape estimation means From the relative shape of the target object and the distance to the target object estimated by the distance estimating means, an absolute shape estimating means for estimating an absolute shape of the target object,
With this configuration, the absolute distance from the image pickup means to the target object is accurately estimated.

[0022]

Embodiments of the present invention will be described below with reference to the drawings. 1 to 9 relate to a configuration of a first embodiment of the present invention, FIG. 1 is an overall configuration diagram of an electronic endoscope system in the first embodiment, and FIG. 2 is a block configuration of an electronic endoscope apparatus. FIG. 3 is a block diagram showing the configuration of the image processing apparatus, FIG. 4 is a block diagram showing the functional configuration of the image processing means configured by the CPU, and FIG. 5 is a configuration of the image processing means configured by the hardware. Block diagram, FIG. 6 is a block diagram showing the configuration of the recording device, FIG. 7 is a block diagram showing a modified example of the input unit of the image processing device, FIG. 8 is a block diagram showing the functional configuration of the relative shape estimating means, and FIG. 9 is It is a block diagram which shows the function structure of a distance estimation means.

In the first embodiment, the absolute shape of an object can be estimated without adding a special device to the endoscope system or changing the structure of the device. This is an example.

As shown in FIG. 1, the endoscope system 1 includes
An electronic endoscope apparatus 2 including an image pickup means, an image processing apparatus 3 that performs image processing that estimates a three-dimensional shape of an object such as a diseased part based on the captured image, and a recording apparatus 4 that records the image.
And a monitor 5 for displaying the image-processed image.

The electronic endoscope apparatus 2 includes an electronic endoscope 6, a light source section 7A for supplying illumination light to the electronic endoscope 6 (see FIG. 2), and a signal processing section 7B for performing signal processing for the image pickup means.
And an observation monitor 8 for displaying an image signal output from the observation device 7.

The electronic endoscope 6 has an elongated insertion part 11 to be inserted into the living body 9, an operation part 12 formed at the rear end of the insertion part 11, and an extension from the operation part 12. The universal cable 13 and the connector 14 provided at the tip of the universal cable 13 can be connected to the observation device 7.

A light guide 15 is inserted into the insertion portion 11 and the connector 14 is connected to the observation device 7, whereby illumination light is supplied from the light source portion 7A to the incident end face as shown in FIG. The illumination light is transmitted by the light guide 15 and is emitted forward from the end surface on the side of the tip portion 16 to illuminate the target site in the living body 9. The illuminated target portion is imaged by the objective lens 17 provided on the tip portion 16 on the CCD 18 arranged at the image forming position, and photoelectrically converted. The objective lens 17 and the CCD 18 form an image pickup section 19 as an image pickup means.

The image signal photoelectrically converted by the CCD 18 is subjected to signal processing by the signal processing section 7B in the observing device 7 to generate an image signal.
And is output to the image processing device 3.

The structures of the light source section 7A and the signal processing section 7B in the observation apparatus 7 are shown in FIG.

The light source section 7A is provided with a lamp 21 which emits light in a wide band from ultraviolet light to infrared light. As the lamp 21, a general xenon lamp, a strobe lamp, or the like can be used. The xenon lamp and the strobe lamp emit a large amount of ultraviolet light and infrared light as well as visible light. Electric power is supplied to the lamp 21 from a power source 22. A rotary filter 28, which is rotationally driven by a motor 23, is arranged in front of the lamp 21. Filters that transmit light in the respective wavelength regions of red (R), green (G), and blue (B) for normal observation are arranged in the rotary filter 28 along the circumferential direction. Further, the motor 23 is a motor driver 25.
The rotation is controlled and driven by.

After passing through the rotary filter 28, R, G,
The light that has been time-sequentially separated into the light in each wavelength region of B is further incident on the incident end of the light guide 15, is guided to the emitting end face on the side of the tip portion 16 via the light guide 15, and the emitting end face is Is emitted to the front to illuminate an observation site or the like.

The return light from the subject (subject) such as the observation site due to the illumination light is transmitted by the objective lens 17 to the CC
An image is formed on D18 and photoelectrically converted. A drive pulse from the driver 31 in the signal processing unit 7B is applied to the CCD 18 via the signal line 26, and an electric signal (image signal) corresponding to the image of the subject photoelectrically converted by the drive pulse is applied. Read and transfer are performed. The image signal read from the CCD 18 is input to the preamplifier 32 provided in the electronic endoscope 6 or the observation device 7 via the signal line 27.

The image signal amplified by the preamplifier 32 is
The signal is input to the process circuit 33, subjected to signal processing such as γ correction and white balance, and converted into a digital signal by the A / D converter 34. This digital image signal is output by the select circuit 35.
For example, it is selectively stored in three memories (1) 36a, memory (2) 36b, and memory (3) 36c corresponding to each color of red (R), green (G), and blue (B). There is. The R, G, B color signals stored in the memory (1) 36a, the memory (2) 36b, and the memory (3) 36c are simultaneously read out, converted into an analog signal by the D / A converter 37, and input. It is output to the observation monitor 8 as R, G, B color signals via the output interface 38,
The observation monitor 8 is adapted to display the observation site in color.

A timing generator 39 is provided in the observing device 7 to generate the timing of the entire system, and the timing generator 39 synchronizes the circuits such as the motor driver 25, the driver 31, and the select circuit 35. Has been.

In this embodiment, the output ends of the memory (1) 36a, the memory (2) 36b, the memory (3) 36c and the synchronization signal output end of the timing generator 39 are connected to the image processing apparatus 3. Further, the image processing device 3 is connected to the monitor 5, and the result of the arithmetic processing by the image processing device 3 is displayed on the monitor 5.

FIG. 3 shows the configuration of the image processing apparatus 3.
The image processing device 3 includes a CPU 40, an information input device 41, and an R.
A main storage device 42 including an AM, an image input interface 43, a display interface 44, a ROM 45, and a recording device interface 46 are provided, and these are connected to each other by a bus.

The information input device 41 is composed of a keyboard and the like, and can input data such as the type of the electronic endoscope 6. The image input interface 43 is
It is connected to the memory (1) 36a, the memory (2) 36b, and the memory (3) 36c, and receives image data from these memories. Further, the display interface 44 is adapted to send image data to be input to the monitor 5.

The CPU 40, as shown in FIG.
An image processing unit 47 that operates according to a program stored in the OM 45 or the like is provided, and the image processing unit 47 moves a tip of the insertion unit 11 of the electronic endoscope 6 on a target object in a body cavity, for example, to thereby perform a plurality of operations. Relative shape estimation means 50 that captures an image and estimates the relative shape of the target object from the stored image, and a position on the target object from the viewpoint of an appropriate position of the tip of the insertion portion 11 of the electronic endoscope 6. Distance estimating means 51 for estimating the distance to the absolute shape estimating means for estimating the absolute shape of the object from the relative shape estimated by the relative shape estimating means and the distance estimated by the distance estimating means. 52, and is comprised.

FIG. 5 shows a block configuration when the arithmetic processing of the image processing apparatus 3 is configured by hardware.

Memory (1) 36a, (2) 36b,
(3) The R, G, and B time-series image data stored in 36c is once input and stored in the memory 61. By inputting the image data stored in the memory 61 to the relative shape estimating means 62 and the distance estimating means 63 and inputting the output results of each to the absolute shape estimating means 64, the absolute shape of the object is calculated and obtained. The obtained three-dimensional image data or two-dimensional image data is stored in the memory 65.

Then, the three-dimensional image data or the two-dimensional image data stored in the memory 65 is read, converted into an analog signal by the D / A converter 66, and output to the monitor 5 via the input / output interface 67. As a result, a three-dimensional or two-dimensional image of the target site is displayed on the monitor 5.

Further, the image processing device 3 is provided with an arithmetic processing controller 68 for controlling the arithmetic operations of the relative shape estimating means 62, the distance estimating means 63 and the absolute shape estimating means 64.
A memory controller 69 for controlling reading and writing of data in the memories 61 and 65, and a selector 70 for selectively outputting image data in the memories 61 and 65 are provided. With this selector 70, the memory controller 69
One of the output terminals of the memory 61 and the memory 65 is selectively connected to the recording device 4 by the control signal, and data can be transferred between the recording device 4 and the memory 61 or the memory 65.

In this embodiment, the image processing device 3 processes the image of the object portion obtained by the electronic endoscope 6 and outputs the processing result to the monitor 5.

FIG. 6 is connected to the image processing device 3 and records image data which is image-processed by the image processing device 3 or records image data which is processed by the image processing device 3 and displayed on the monitor 5. 3 illustrates a more specific configuration example of the recording device 4.

Input of image data to the recording device 4
Not only the electronic endoscope device 2, but also an input device such as an analog or digital magneto-optical disk, a VTR, an optical disk, a moving image memory can be used.

An image recording device 80 provided in the recording device 4.
Is an analog type image recording device such as a VTR or an analog magneto-optical disk device, the output of which is A / D converted by an A / D converter 81 and then input to the image processing device 3 as a digital image. Further, it is possible to input the processing result image by the image processing device 3 via the D / A converter 84 and record it in the image recording device 80.

The moving image file device 82 is a digital type moving image file device including a semiconductor memory, a hard disk, a digital magneto-optical disk device, and the like.
The output is input to the image processing device 3 as a digital image. Further, the still image filing device 83 has a semiconductor memory, a hard disk,
It is a digital still image filing device composed of a digital magneto-optical disk device or the like, and its output is input to the image processing device 3 as a digital image. In addition, the processing result image by the image processing device 3 is displayed as a moving image file device 82.
It is also possible to record in the still image file device 83.

In FIG. 2, the image signal photoelectrically converted by the CCD 18 becomes an image signal signal-processed by the signal processing unit 7B in the observation device 7, and is output to the observation monitor 8 and the image processing device 3. To be done.

The two-dimensional image data input to the image processing device 3 is once sent to the recording device 4 and the image signal is recorded. Further, the recorded image signal is sent again to the image processing device 3, where the process of estimating the three-dimensional shape of the subject is performed, and is sent to the recording device 4 and the monitor 5, where the estimated three-dimensional shape is recorded and recorded. The display is done.

Further, as shown in FIG. 7, the image processing apparatus 3 has an A / D in front of the image input interface 43.
The converters 90a, 90b, 90c may be provided. Thereby, R, G, B analog signals from an external device, for example, the input / output interface 38 of the observation device 7
Analog signals R, G, B from the A / D converter 90a,
90b, 90c, and A / D converters 90a, 90
By recording the digital signals obtained in steps b and 90c in the RAM 42, it is possible to obtain digital R, G, B image data.

The relative shape estimating means 50, as shown in FIG. 8, is a position detecting means 101 for detecting the position of the same point in an image obtained by picking up the same target object from a plurality of positions by the image pickup means, and the position detecting means 101. The motion estimating means 102 estimates the relative movement of the image pickup means from the position of the same point detected by the detecting means 101, and the position of the same point detected by the position detecting means 101 and the motion estimating means 102. A shape estimation unit 103 that estimates the relative shape of the target object based on the detected movement of the imaging unit.

As shown in FIG. 9, the distance estimating means 51 estimates the frequency characteristic in the proper region including the point at the same point of each image detected by the position detecting means 101. The means 104, the image detecting means 105 for detecting an image in which the frequency characteristic of each image spreads to the highest frequency region, and the image detection position of the image detected by the image detecting means 105 are determined to be most focused, Absolute distance estimating means 106 for estimating an absolute distance by making the distance from the viewpoint of the means to a point on the target object the same as the focal length of the imaging means.

The absolute shape estimating means 52 outputs the relative shape of the target object as the output result of the relative shape estimating means 50 and the output result of the distance estimating means 51 as the absolute shape from the image pickup means to a point on the target object. It has a function of estimating the absolute shape of the target object based on the distance.

Next, referring to FIGS. 10 to 17, the first
The operation of the image processing apparatus according to the embodiment will be described. FIG. 10 is a flowchart showing the overall flow of processing for estimating the relative shape of the target object, FIG. 11 is an operation explanatory view for explaining each processing of FIG. 10, and FIG. 12 is an image obtained in time series. FIG. 13 is an explanatory view showing how a shift map is created from data, FIG. 13 is an explanatory view showing a coordinate system of central projection, FIG. 14 is an explanatory view showing a relation between a rotation vector and a translation vector related to the motion of the image pickup means, and FIG. FIG. 16 is an explanatory diagram showing a change in distance to the target object due to movement of the means, FIG. 16 is an explanatory diagram showing how to obtain frequency characteristics of each image obtained in time series, and FIG. 17 is a characteristic showing frequency characteristics of image data. It is a figure.

The endoscopic images obtained by the electronic endoscope 6 are sequentially sent out as image signals by the observation device 7 to the image processing device 3. The image processing device 3 sends the image signals of the endoscopic images sequentially sent to the recording device 4 for recording in an arbitrary period. Then, the image processing device 3 inputs the image data of a plurality of frames of the endoscopic image signal (two-dimensional image data) recorded in the recording device 4, and detects the position of the same point from the image data of the plurality of frames. After detecting, the shift map representing the movement of the detected position is calculated.

Thereafter, the motion vector of the image pickup means (motion vector of the tip of the electronic endoscope 6) is estimated from the obtained shift map. Further, a relative distance between the image pickup device and the target object is calculated by using the estimated motion vector of the image pickup device and the shift map between the respective images, and the absolute shape of the target object is estimated.

The three-dimensional data of the obtained absolute shape is displayed as a two-dimensional image or a three-dimensional image when viewed from the direction of an arbitrary image pickup means (wire frame or the like).
Create image data. The three-dimensional image data or the two-dimensional image data created by the image processing device 3 is displayed on the monitor 5 and simultaneously recorded on the recording device 4.

First, a method for estimating the relative shape of the object will be described. FIG. 10 is a flowchart showing the overall flow of processing for estimating the relative shape, and FIG. 11 is shown in FIG.
It is an explanatory view of an operation for explaining each processing of 0. Hereinafter, each process will be sequentially described.

(1) Calculation of shift map First, at step S1, an image is taken while moving the image pickup means by the electronic endoscope 6, and the two-dimensional image data picked up in time series is used as an image processing apparatus. Enter in 3. Then, in step S2, the same point between a plurality of images is detected and a shift map is created.

At this time, as shown in FIG.
A time-series image of the same target object is captured by moving the tip of the endoscope. As a result, time-series images f0 to fn are obtained corresponding to the movement loci of the image pickup means indicated by P0 to Pn.

Then, as shown in FIG. 12B, a plurality of points Pab (a = 0, 1, ..., Q, b = 0, 1) are added to the first image f0 of the captured time series images. , ..., r) and the template matching method (reference: computer image processing introduction: Soken Publishing Co., Ltd. supervised by Hideyuki Tamura: p.148-p.150).
The position of the same point as the point Pab set in 1, 2, ..., N) is detected.

By detecting the position of the same point in each image, the remaining image f with respect to the point Pab set in the image f0 is detected.
The positional deviation from i (i = 1, 2, ..., N), that is,
The shift map Si (i) which is the amount of movement of the point Pab for each image
, 1, 2, ..., N) are obtained.

The shift map for the image fi is expressed as the amount of movement in the x and y directions as follows.

[0064] The template matching method may be the color matching method disclosed in USP 4,962,540 by the same applicant.

(2) Estimation of Movement Next, in step S3, the movement of the image pickup means (movement of the endoscope distal end portion 16) is estimated from the obtained shift map.

FIG. 13 shows the coordinate system of the central projection. The image plane is placed perpendicular to the Z axis at a distance e from the image origin O. The point (X, Y, Z) in space is this point (X, Y, Z
It is assumed that the image is projected at a point where a straight line connecting (Y, Z) and the origin O intersects with the image plane. Assuming that the coordinates of the intersection on the image are (x, y), x, y are

[Equation 1] Becomes

By the movement of the image pickup means, a point X in space =
When (X, Y, Z) moves to X '= (X', Y ', Z'), the motion of the image pickup means at that time is a rotation vector R representing rotation about the visual axis passing through the origin, Translation vector h =
It is expressed as follows using (hx, hy, hz).

[0068]

[Equation 2] Here, when the points X and X ′ are represented using the distances r and r ′ from the origin and the unit direction vector m (N vector), respectively,

(Equation 3) Becomes However,

(Equation 4) It is. Therefore, using equation (4), equation (2) becomes

(Equation 5) Becomes

Here, as shown in FIGS. 14A and 14B, since the three vectors m, Rm ', and h are on the same plane in space, their scalar triple product is 0. Becomes

[0070]

(Equation 6) When the same point between images is detected by the template matching, the shift map may include an error due to a detection error. Therefore, in this example, using the method of least squares, for a plurality of points set in the image,

(Equation 7) A rotation vector R and a translation vector h are obtained so as to satisfy However, the estimated translation vector h is estimated as a unit vector h '.

(3) Calculation of Relative Distance between Imaging Device and Target Object Next, in step S4, a relative distance from the imaging device to the target object is calculated from the estimated motion of the imaging device.

Taking the inner product of both sides of equation (5) and m and Rm ',

(Equation 8) Becomes If this is solved, the distance to each point can be obtained as follows.

[0073]

[Equation 9] However, since the translation vector h is estimated as the unit vector h ', r and r'are determined by an appropriate magnification k as in the following equation.

[0074]

## EQU10 ## r = kr (10)

[Mathematical formula-see original document] Therefore, from the position of the image pickup means when the image f0 is picked up, the image pickup means of each image fi (i = 1, 2, ..., N) is picked up. Each relative movement to the position (R
i, hi (i = 1, 2, ..., N)) are estimated.

Based on the relative motion estimated for each image fi (i = 1, 2, ..., N) and the shift map, the relative distance from the position of the image pickup means to the target object for each image fi. To estimate. This relative distance corresponds to the relative shape when the target object is observed from the image pickup means.

Next, a method of obtaining one absolute value from the obtained relative shape of the object will be described.

When the images of the same target object are picked up in time series by moving the tip of the endoscope, a defocused image is picked up according to the distance between the tip of the endoscope and the target object.

The defocused image includes the image of the rear blur generated when the distance between the tip of the endoscope and the target object is longer than the focal length of the endoscope, and the image of the front blur generated when the distance is close. There is an image.

In the time-series images including such front and rear blurred images, the images captured when the distance between the endoscope tip and the target object and the focal length of the electronic endoscope match each other are obtained. included.

Then, by detecting the most in-focus image and the most in-focus point in the image from the time series images, the focal length of the electronic endoscope is determined to be the in-focus state. By setting the distance from the position where the detected image is captured to an appropriate position on the target object, it is possible to estimate an absolute value of the distance between the image capturing unit and the target object.

In the present embodiment, after the same point is detected in the process of estimating the relative shape described above, a plurality of points Pab (a = 0, 1, ..., Q, b = 0, set in the image f0 are set.
1, ..., r), an appropriate point is selected, and the frequency characteristic is obtained for that point and the same point for each image.

When a time-series image is picked up by moving the image pickup means, the size of the target object in each picked-up image changes according to the position of the image pickup means. For this reason, when the frequency characteristic of each image is obtained from the size of one region, the frequency characteristic cannot be directly compared because the scaling factor is included in addition to the blur.

Therefore, since the relative distance between the position of the image pickup means and the position of the target object is estimated for each image when the relative shape is estimated, as shown in FIGS. 15 and 16, respectively. The size of the area is determined according to the ratio of the distances, and the frequency characteristics of the images are compared.

As shown in FIG. 15, the ratio of the distance from the image pickup means between the point Pab set in the image f0 and the same point P'ab in the image fi is given by the equations (10) and (11).

[Equation 12] d0: di = kr: kr '= r0: ri (12)

Further, as shown in FIG. 16, if the area W0 including the point Pab set in the image f0 is set as W0x and W0y,
The area Wi containing the same point in the image fi is

(Equation 13) Becomes

Then, the area Wi including the same point of the image fi
Area W i is enlarged or reduced so that the area W i has the same size as the area W 0 including the point Pab set in the image f 0. At this time, the magnification K due to enlargement or reduction is

[Equation 14] It is.

In this way, the size of the area is set for each image, the Fourier transform is performed on the partial image Ipi obtained by enlarging or reducing the set area, and the frequency characteristic of each partial image is calculated. Ask.

Here, in order to detect an image in which the frequency characteristic of the partial image Ipi is most wide, as shown in FIG. 17, appropriate regions λ0 and λ1 are set in the high frequency portion of each frequency characteristic, and The average power Pavr in the frequency domain surrounded by λ0 and λ1 is

(Equation 15) The total power Psum in the frequency domain enclosed by λ0 and λ1
To

(Equation 16) Then, the partial image Ipi that maximizes this Pavr or Psum is obtained.

As a result, the images f taken in time series are obtained.
Of 0 to fn, the point Pab set in the image f0 and the image including the most focused point among the same points in the image fi are detected.

The relative shape estimation means estimates the distance from the position of the image pickup means when the image is picked up to the position of the target object (the most focused point) as the focal length of the electronic endoscope. The relative distance from the image capturing means to one point on the target object can be represented as an absolute distance,
Based on this, it is possible to estimate the absolute value shape of the target object.

In the present embodiment, the absolute value from one point of the image f0 is estimated for the distance from the image pickup means to the target object, but the absolute value is estimated for a plurality of points and estimated. The absolute shape of the target object may be estimated from the average value of the ratio and the relative shape.

As described above, according to the present embodiment, characteristics of images obtained in time series without adding a special device to the conventional endoscope system or changing the device configuration. Can be used to estimate the absolute shape of the target object.

Further, in the present embodiment, the tip of the electronic endoscope is moved as shown in FIG. 12 to create a front blur and a rear blur in a time-series image of the object. Since movements in the living body such as heartbeats and respirations occur, it is possible to easily capture a plurality of images in which the states of front blur and rear blur are generated without moving the electronic endoscope as shown in FIG. it can.

Next, a second embodiment of the present invention will be described with reference to FIGS.

The second embodiment is an example in which the functional structure of the distance estimating means is changed, and the parts other than the distance estimating means have the same structure as in the first embodiment.

The distance estimation means of the second embodiment is a halation detection means for detecting an image in which halation exists and a position of halation in the image with respect to a plurality of time-sequentially imaged images. The normal vector estimating means for estimating the normal vector of the tangent plane at the halation position from the halation detecting means with respect to the shape estimated by the shape estimating means, and the normal vector of the tangent plane and the relative shape estimating means. Absolute distance estimating means for estimating the distance from the viewpoint to a specific position of the target object from the relationship between the estimated direction vector from the viewpoint to the target object and the distance between the center of the outlet of the light source and the lens of the imaging means,
Is configured.

When the endoscope tip is moved with respect to the same target object to capture a time-series image, halation generally occurs in the image according to the distance between the endoscope tip and the target object. There are many. On the surface of the target object in which halation occurs, as shown in FIG. 18, the light emitted from the tip of the endoscope and the light reflected from the target object are incident at the same angle with respect to the normal line of the surface of the target object. Or reflect.

Therefore, if the relative shape of the target object and the position of the halation are known, the distance from the position where the image in which the halation is generated to the halation position on the target object can be given as an absolute value. it can.

The operation for estimating the absolute distance from the image pickup means to the target object in the second embodiment will be described below.

First, as shown in the first embodiment, the relative shape estimation means estimates the relative shape of the target object.

After that, the position of halation is detected for the plurality of time series images obtained by the image pickup means. As a method of detecting halation, a method of detecting as halation when the brightness level of each pixel of each image exceeds a specific value (threshold value) by threshold processing is used.

Next, as shown in FIG. 18, the normal vector estimating means obtains a vector m from the viewpoint O to the halation position P of the target object and a normal vector n of the tangent plane at the halation position P. However, the vector m and the normal vector n are N vectors whose unit vector is a set of normalized simultaneous coordinate systems.

Here, as shown in FIG. 19, the intersection point between the straight line extending the normal vector n of the tangential plane at the point P on the object and the straight line connecting the light source S from the viewpoint O is Q, and the point P is the point. The vector to Q is N, the vector from point O to point P is M,
If the vector from the viewpoint O to the light source S is d,

[Equation 17] Becomes Also, these M and N are given by appropriate coefficients k0 and k1,

(Equation 18) And each of them is substituted into the equation (17),

[Equation 19] Becomes

When the outer product of the normal vector n is calculated on both sides of the equation (19),

(Equation 20) Since the denominator and the numerator of equation (20) are vectors, the coefficient k0 can be obtained by taking the n × m inner product of the denominator and the numerator.
Ask for.

[0105]

(Equation 21) Since the vectors m, n, and d in this equation (21) are all known vectors, the coefficient k0 is obtained, and the distance from the image pickup means to the position where halation is occurring on the target object is an absolute value. Obtainable.

Therefore, since the absolute one value estimated by halation and the relative distance from the image pickup means to the target object by the relative shape estimation means are estimated, the absolute distance to the target object is calculated. Given this, the shape of the target object can be estimated as an absolute value based on this.

An example of the method of estimating the normal vector will be shown. As shown in FIG. 20, the halation position I on the image
Points IA, IB, and IC close to H are extracted, and the points on the target object corresponding to the respective points are detected as PH, PA, PB, and PC.
Then, the plane composed of three points PA, PB, and PC is

[Expression 22] lx + my + nz + p = 0 (22) Let the normal vector (l, m, n) of this plane be the normal vector n of the tangent plane at the halation position PH on the object.

Further, the halation position PH on the target object
It is also possible to estimate a curved surface near the halation position PH from a plurality of points close to and to estimate the normal vector of the halation position on the curved surface (Reference: Shape Processing Engineering by Computer Display [I] Nikkan Kogyo Shimbun). Fujio Yamaguchi: p.49-p.66).

In this embodiment, one absolute value regarding the distance between the image pickup means and the target object is estimated from the halation of one image. However, a plurality of halations of one image or a plurality of halations of a plurality of images are estimated. The absolute shape of the target object may be estimated from the average value of the ratio and the relative shape by estimating the absolute value of each from the ratio of the estimated value and the relative value. .

As described above, according to the present embodiment, characteristics of images obtained in time series without adding a special device to the conventional endoscope system or changing the device configuration. Can be used to estimate the absolute shape of the target object.

Further, in general, when a plurality of endoscopic images are picked up in time series, halation is often included in the plurality of images. Therefore, in order to obtain halation, the tip of the electronic endoscope is special. The absolute shape of the object can be easily estimated without the need to move.

Next, a third embodiment of the present invention will be described with reference to FIGS.

The third embodiment makes it possible to estimate the absolute shape of an object by adding an additional device to the endoscope system without changing the structure of the device. 3 shows an example of the configuration.

As shown in FIG. 21, in addition to the configuration of the endoscope system of the first embodiment, a laser generator 110 for generating a laser beam, and a laser beam from the laser generator 110 are supplied to an electronic endoscope. 6, a probe 111 for irradiating a target object by inserting from a forceps port 112 of 6,
The image processing means 47 in the image processing device 3 is configured to have a distance estimating means for estimating a distance from a viewpoint to a specific point of the target object from a plurality of images taken in a state where the laser light is irradiated. It

The distance estimating means of the third embodiment is an emitting position vector calculating means for calculating an emitting position vector from the viewpoint of the image pickup means to a position where the laser light is emitted, the calculated emitting position vector and the aforesaid Laser irradiation vector calculation means for calculating a laser irradiation vector composed of the irradiation position vector of the irradiation position of the laser light irradiated on the target object from the viewpoint of the image pickup means, and the outer product of the calculated laser light irradiation vector From 0 to 0, the absolute distance estimating means for estimating the distance from the image pickup means to the position on the target object irradiated with the laser beam is provided.

The operation of this embodiment will be described below. FIG.
As shown in FIG. 1, the probe 111 connected to the laser generator 110 is inserted from the forceps port 112 of the electronic endoscope 6. Then, the probe 11 inserted from the forceps port 112
The tip of No. 1 is projected from the tip of the endoscope so that the tip of the No. 1 exists at a specific position of the observed image on the monitor 8. In this state, while irradiating the laser beam, the tip of the electronic endoscope 6 is moved over the target object to capture a plurality of images.

The plurality of picked-up images are taken into the image processing apparatus 3, and the relative shape estimation means 50 described above estimates the relative shape of the target object.

Further, since the tip of the probe projects from the tip of the endoscope so that it exists at a specific position of the observed image on the monitor 8, the position of the image pickup means of the electronic endoscope 6 and the tip of the probe are different. The positional relationship can be obtained.

Therefore, the distance to the irradiation position of the laser light on the target object can be estimated from the tip position of the probe, the irradiation position of the laser light on the observed image, and the relative shape of the target object.

A method of estimating the distance to the irradiation position of the laser light on the target object by the distance estimating means will be described below with reference to FIG.

FIG. 22A shows the tip of the electronic endoscope 6 and the target object when the target object is imaged at two different positions.
And the irradiation position of the laser beam.

Since the position of the probe tip and the irradiation direction of the laser beam with respect to the position of the image pickup means are the same no matter how they move on the target object, they are set at two different positions in FIG. 22 (a). When the image pickup means at the time of image pickup is overlapped on the same point, it can be expressed as shown in FIG. Here, FIG. 22B is represented by a vector as shown in FIG.

L is a vector from the viewpoint of the imaging means to the tip of the probe, and M1 and M2 are vectors from the viewpoint of the imaging means to the irradiation position of the laser light radiated on the target object when imaging is performed at two different positions. Then,

(Equation 23) Becomes

Here, m1 and m2 are unit direction vectors to the irradiation position of the laser light irradiated onto the target object from the viewpoint of the image pickup means estimated by the relative shape estimation means, k
Is the magnification.

Further, as shown in FIG. 22C, since the irradiation direction of the laser light is always constant, the vector (L-km1) obtained from L and m1 and the vector (L-m1) obtained from L and m2. -Km2) is a vector with the same direction but different magnitude. Therefore, the outer product of these two vectors is 0 as shown in the following equation.

[0126]

(Equation 24) When the coefficient k is calculated from this equation (24),

(Equation 25)

(Equation 26) Becomes

Since the denominator and the numerator of the equation (26) are vectors, if the inner product of m1 × m2 is taken for the denominator and the numerator,

[Equation 27] Becomes

The vector L, m1, m2 of this equation (27)
Is a known vector, the coefficient k is obtained, and the distance from the image pickup means to one point on the target object can be obtained as an absolute value.

Further, in the equation (25),

[Equation 28] Then, equation (25) becomes

(Equation 29) Then, by squaring both sides of equation (29),

[Equation 30] Is required (however, · represents the dot product).

From a plurality of images, there are a plurality of vectors from the viewpoint of the image pickup means to the irradiation position of the laser light irradiated on the target object.

Therefore, the least squares method is applied to the equation (30),

(Equation 31) A method for estimating k that satisfies

The procedure for calculating the vector L from the viewpoint of the image pickup means to the tip of the probe will be described below.

As shown in FIG. 22A, the relationship between L-km1 and L-km2 always holds regardless of the value of k.
That is, these are vectors having the same direction but different magnitudes regardless of any value of k other than 0.
Therefore, if k = 1,

(Equation 32) And obtain L.

Equation (32) is

[Equation 33] And each vector is expressed as a component,

(Equation 34) Becomes

Substituting equation (34) into equation (33),

(Equation 35) Expanding this equation (35),

Equation 36] m1ym2z-m1zmy - {Ly (m1z + m2z) -Lz (m1y + my)} = 0 m1zm2x-m1xmz - {Lz (m1x + m2x) -Lx (m1z + mz)} = 0 m1xm2y-m1ymx - {Lx (m1y + m2y) -Ly ( m1x + mx)} = 0 (36) and Lx, Ly, and Lz can be obtained from the equation (36).

It is possible to estimate the absolute value shape of the target object based on the absolute distance from the image pickup means thus obtained to one point on the target object.

As described above, according to this embodiment, an additional device (here, a laser light irradiation device) is added without changing the configuration of the conventional endoscope system. The absolute shape of the object can be estimated. At this time, even if the irradiation direction of the laser light emitted from the probe changes every time the probe is inserted from the forceps port of the endoscope, the absolute shape can be always estimated. However, in this case, it is assumed that the irradiation direction of the laser light does not change while the distal end of the endoscope is moved to obtain a plurality of images.

Next, a fourth embodiment of the present invention will be described with reference to FIGS.

The fourth embodiment, like the first and second embodiments, does not add a special device to the endoscope system or changes the structure of the device so that the absolute object can be detected. It is an example of a configuration that makes it possible to estimate a typical shape.

The fourth embodiment is an example in which the functional structure of the distance estimating means is changed, and the parts other than the distance estimating means have the same structure as in the first embodiment.

The distance estimating means of the fourth embodiment includes a normal vector estimating means for estimating a normal vector of the surface of the target object, and a light source position vector estimating means for estimating a vector from the viewpoint of the image pickup means to the light source. A luminance for estimating the luminance of a point on the target object from the relationship between the relative distance from the viewpoint of the image pickup means estimated by the relative shape estimation means to the point on the target object and the normal vector and the light source position vector Estimating means, equation calculating means for obtaining the ratio of luminance of a plurality of points on the target object from the luminance estimating means and an actual image, and calculating a plurality of equations relating to the luminance ratio, and a plurality of equations relating to the luminance ratio. And an absolute distance estimating means for estimating the absolute distance to the target object by estimating the magnification of the vector from the image pickup means to the point on the target object.

As shown in FIG. 23, assuming that the illumination light of the monocular endoscope is a point light source and the surface of the target object is a perfect diffusing surface, the brightness value E at one point on the target object obtained by the image pickup means.
Is

(37) Is required. Here, ρ is the diffuse reflectance, x is the distance from the light source to one point on the target object, and θ is the angle formed by the vector from the light source to one point on the target object and the normal vector at that point.

As shown in FIG. 24, the angle θi formed by the vector Xi from the light source to an appropriate point on the target object and the normal vector ni of the target object at one point is

(38) Becomes Since the normal vector ni is a unit direction vector,

[Equation 39] It is.

From equations (37) and (38), the luminance Ei is

(Equation 40) Becomes

As shown in FIG. 25, from the viewpoint O of the image pickup means obtained on the basis of the position of the image pickup means when the image fi is picked up by the relative shape estimation means, two points on the target object are detected.
The vectors M1 and M2 to the points A and B are

[Equation 41] Becomes However, k is a magnification.

When the light source position vector from the viewpoint O of the image pickup means to the light source S is L, the vectors X1 and X2 from the two points of the target object to the light source are

(Equation 42) From equation (40), the brightness at each point E1, E
2 is

[Equation 43] Is estimated as.

Further, assuming that the luminances of two points obtained from the actual image fi are I1 and I2, respectively, the ratio of these luminances is

[Equation 44] Becomes

By solving the equation (44) for k, a plurality of solutions are calculated. Therefore, the luminance ratio is calculated from at least another set of different points, and the magnification k that satisfies each equation is estimated.

An example of the method of estimating the light source position vector will be shown. As shown in FIG. 26, the viewpoint O is perpendicular to the optical axis.
A mirror and a grid pattern with a known interval are placed on the mirror surface at a distance Z from and an image of the mirror pattern is captured. Assuming that the distance between the lattice patterns on the mirror surface is d, the distance between the lattice patterns on the captured image is d ′, and the focal length is e, the distance Z from the viewpoint O to the mirror surface is

[Equation 45] Becomes

As shown in FIG. 27, the reflection position vector V (X, Y, Z) on the mirror surface from the reflection position (x, y, e) of the illumination light source in the captured image is

[Equation 46] Becomes

The distance b between the irradiation end of the illumination light source and the mirror surface
Is measured, and a symmetrical position vector V '(X', Y ', Z') with respect to the mirror surface of the irradiation end of the illumination light source is obtained,

[Equation 47] Becomes

Therefore, the irradiation position vector VS of the irradiation light source is

[Equation 48] Therefore, the light source position vector can be obtained from this equation (48).

The estimation of the normal vector can be performed by using the method shown in the second embodiment.

Further, the distance estimating means of the present embodiment makes it possible to estimate the vector magnification from the image pickup means to the point on the target object from three points in the image. It is also possible to estimate the magnification.

In this case, the least squares method is applied to the equation (44), and the luminance ratio of each point is

[Equation 49] It can be estimated by determining the magnification k that satisfies the above condition.

In the relative shape estimating means shown in FIG.
Also, as shown in FIG. 11, the position at which each image of the images obtained in time series is captured is set as the viewpoint of the image capturing unit, and the relative distance from the viewpoint to the target object is calculated.

In the distance estimating means described above, one magnification k is estimated from one relative shape estimated by the relative shape estimating means and the image.

However, since the relative shape estimating means obtains the relative distances between the plurality of images and the positions where the respective images are picked up, the relative shape estimating means determines the relative shapes and their relative distances. It is also possible to obtain the magnification kx from the image and estimate one magnification <k> from the average value as shown in the following equation.

[0159]

[Equation 50] As described above, according to the present embodiment, images are obtained in time series without adding a special device to the conventional endoscope system or changing the device configuration, and the images are It is possible to estimate the absolute shape of the target object by calculating one absolute value from the relative distance from the viewpoint to the target object, using the imaged position as the viewpoint of the imaging unit.

Next, a fifth embodiment of the present invention will be described.

The fifth embodiment is an example in which the same color point detecting means for detecting points of the same color in an image is added to the configuration of the fourth embodiment.

In the fourth embodiment, as shown in FIG. 25, two points at different positions are extracted and the ratio of the brightness of the two points and the ratio of the brightness of another two sets of points are used to determine the target object. Estimated the absolute distance of.

In the present embodiment, points of the same color on the target object are considered to have the same diffuse reflectance, and points of the same color in the captured image are extracted to obtain the absolute distance.

In order to extract points of the same color in the picked-up image, the color components representing each point in the image are Rj, Gj, Bj.
And when the color components of the reference point are R0, G0, B0,

[Equation 51] R0 = Rj (j ≠ 0) G0 = Gj (j ≠ 0) (51) B0 = Bj (j ≠ 0) Choose. Alternatively, set an appropriate range as in the following equation,
There is also a way to choose points with colors within that range.

[0165]

(R0-α) ≤Rj≤ (R0 + α) (j ≠ 0) (G0-β) ≤Gj≤ (G0 + β) (j ≠ 0) (52) (B0-γ) ≤Bj≤ (B0 + γ) (j ≠ 0) In order to extract the same color even when the brightness changes, the value obtained by normalizing the components at each point

(Equation 53) There is a method of extracting a point where each color component is the same or within an appropriate range by using.

As a method of estimating the vector magnification from the image pickup means to a point on the target object, the difference in color between two points is used as a weighting function wij, and the magnification is estimated from a plurality of points in the image. Can also be considered.

For example, a function obtained by subtracting a value obtained by multiplying the sum of squares of the difference in luminance between the two color components by an appropriate coefficient Wk.

(Equation 54) Is a weighting function wij. However, Wk is

[Equation 55] The condition of is satisfied.

From this, the weighting function wij is applied to the least squares method of the equation (49), and the ratio of the brightness of each point is

[Equation 56] It can be estimated by determining the magnification k that satisfies the above condition.

According to the present embodiment, points of the same color on the target object are considered to have the same diffuse reflectance, and the brightness ratio is calculated using the points having the same diffuse reflectance to obtain the target object. You can accurately estimate the absolute distance to. Therefore, as in the fourth embodiment, it is possible to accurately estimate the absolute shape of the target object without adding a special device or changing the structure of the device.

Next, a sixth embodiment of the present invention will be described with reference to FIGS. 28 to 32.

The sixth embodiment is an example in which peak detecting means for detecting a point at which a peak of cross-correlation at each point in an image becomes sharp is added to the configuration of the fourth embodiment.

In the present embodiment, an appropriate one image is extracted from a plurality of images obtained in time series, and a specific area in which a clear pattern or a fine pattern exists is used as a template image. To do. If the corresponding points of this template image and other images are detected, the corresponding points can be accurately detected.

At the point where the corresponding point is accurately detected, it is considered that the relative distance and the normal vector corresponding to the point are also accurately estimated.

Therefore, in this embodiment, a point in which the corresponding points are detected with high accuracy is selected, and the point is used as a template image in the distance estimation method in the fourth embodiment to determine the absolute distance to the target object. Ask.

As shown in FIG. 28, when an appropriate area in a certain image A is used as a template image and a specific area in another image B is moved to obtain a cross-correlation value for each point, the template image is obtained. Figure 2
A mountain-shaped characteristic diagram such as the bird's-eye view shown in FIG. 9 is obtained.

As shown in FIG. 30, the value of this cross-correlation shows that the characteristic diagram shows a sharp shape when the template image is an area where a clear pattern or a fine pattern exists in the image.
Further, in the case of an unclear pattern or a blurred image, the characteristic diagram shows a broad shape.

Therefore, a point having a sharp cross-correlation value can be set as a point at which the corresponding point can be accurately detected.

Therefore, in the present embodiment, as shown in FIG. 31, a plurality of points Pab (a = 0, 1, 2, ..., Q, b = 0, in the image f0 set by the relative shape estimating means are set. 1,
2, ..., R), an appropriate area Tab (hereinafter referred to as a template image) is set around each point. Then, the template image Tab is moved to an appropriate area Sab (hereinafter referred to as a search area) in another image fi, and the value of the cross-correlation for each point is obtained.

As the value of the cross correlation,

[Equation 57] A normalized cross correlation such as However, ∬S is the integral of the search area Sab, <fi> is the average value in the area equivalent to the template image in the search area, and <t> is the average value of the template image.

Next, as shown in FIG. 32, a bird's-eye view of cross-correlation values is cut out in the x and y directions centering on the maximum position, and the maximum cross-correlation value is set to Dp and the cross-correlation value threshold value is set. Let Dx be the spread in the x and y directions as Wx,
Wy.

At this time, the sharpness of the shape of the cross-correlation value is

[Equation 58] Then, the larger the value of Expression (58), the sharper the shape.

As described above, the sharpness of the shape relating to the value of the cross-correlation is obtained with respect to the points set in the image f0, and the points having large sharpness are extracted by the threshold processing or the like.

Using the point where the peak of this cross-correlation becomes sharp, the absolute distance to the target object is calculated according to the distance estimation method of the fourth embodiment.

As a method of estimating the magnification of the vector from the image pickup means to the point on the target object, the sharpness of the shape of the cross-correlation value is used as the weighting function wij, and the magnification is estimated from a plurality of points in the image. Can also be considered.

For example, the value Sh of the two points having the smaller shape sharpness multiplied by an appropriate coefficient Wk

## EQU00005 ## Wij = Sh.times.Wk (i.noteq.j) (59) is a weighting function wij.

From this, the weighting function wij is applied to the least squares method of the equation (49), and the ratio of the luminance of each point is

[Equation 60] It can be estimated by determining the magnification k that satisfies the above condition.

According to the present embodiment, the absolute distance to the target object can be accurately estimated by calculating the luminance ratio using the point where the normal vector is accurately estimated.
Therefore, as in the fourth embodiment, it is possible to accurately estimate the absolute shape of the target object without adding a special device or changing the structure of the device.

Next, a seventh embodiment of the present invention will be described.

The seventh embodiment is an example in which the functional configuration of the distance estimating means of the fourth embodiment is changed.

The distance estimating means of the seventh embodiment includes a normal vector estimating means for estimating a normal vector of the surface of the target object, and a light source position vector estimating means for estimating a vector from the viewpoint of the image pickup means to the light source. A luminance for estimating the luminance of a point on the target object from the relationship between the relative distance from the viewpoint of the image pickup means estimated by the relative shape estimation means to the point on the target object and the normal vector and the light source position vector The estimating means, the luminance normalizing means for normalizing the luminance obtained by the luminance estimating means for each point in the image and the luminance obtained from the image by the average of the surrounding luminance, and the luminance normalizing means. Absolute distance estimation means for estimating the absolute distance to the target object by estimating the magnification of the vector from the image pickup means to the point on the target object by the least-squares method based on the obtained normalized luminance of each point When Configured to have a.

In the fourth embodiment, as shown in FIG. 25, two points at different positions are extracted, and the ratio of the brightness of the two points and the ratio of the brightness of another two sets of points are used to extract the target object. Estimated the absolute distance of.

In the present embodiment, the brightness of a point selected from a plurality of points on the target object is normalized by the average of the surrounding brightness to suppress the influence of noise or the like and to reach the target object. The desired distance.

A method of estimating the vector magnification from the image pickup means to the point on the target object in the present embodiment will be described.

The brightness of an appropriate point in the image fi is I (X,
Y) and the peripheral luminance is I (x, y), the value I ′ (X, Y) obtained by normalizing the luminance I (X, Y) by the average of the peripheral luminance is

[Equation 61] Becomes

From equations (43) and (61), I '(X,
Y) is

(Equation 62) Therefore, it can be seen from Expression (62) that Expression (61) is not affected by the diffuse reflectance ρ.

In addition, the brightness estimated from the relative shape is Im
Assuming that (X, Y) and the peripheral brightness thereof be Im (x, y), a value Im '(X, Y) obtained by normalizing the brightness Im (X, Y) by the average of the peripheral brightness is

[Equation 63] Becomes

From equations (43) and (63), Im '(X,
Y) is

[Equation 64] Becomes

From equations (62) and (64), if the luminance value in the image is not affected by noise, I '(X, Y)
And Im '(X, Y) are equal.

Actually, I'due to an error such as noise.
Since the values of (X, Y) and Im '(X, Y) are not equal, the least squares method

[Equation 65] A scaling factor k that satisfies the above is estimated from a plurality of normalized luminances.

According to this embodiment, the influence of noise or the like can be suppressed by normalizing the luminance obtained from the image and the relative shape with the average of the luminance of the surroundings. Accurate distance can be estimated.
Therefore, as in the fourth embodiment, it is possible to accurately estimate the absolute shape of the target object without adding a special device or changing the structure of the device.

Next, an eighth embodiment of the present invention will be described with reference to FIG.

The eighth embodiment is an example in which the functional configuration of the distance estimating means of the fourth embodiment is changed.

The distance estimating means of the eighth embodiment includes a normal vector estimating means for estimating a normal vector of the surface of the target object, and a light source position vector estimating means for estimating a vector from the viewpoint of the image pickup means to the light source. Luminance for estimating the luminance of a point on the target object from the relationship between the relative distance from the viewpoint of the image pickup means estimated by the relative shape estimation means to a point on the target object and the normal vector and the light source position vector Estimating means, equation calculating means for obtaining the ratio of luminance of a plurality of points on the target object from the luminance estimating means and an actual image, and calculating a plurality of equations relating to the luminance ratio, and a plurality of equations relating to the luminance ratio. And an absolute distance estimating means for estimating the absolute distance to the target object by estimating the magnification of the vector from the image pickup means to the point on the target object.

In the fourth embodiment, as shown in FIG. 25, two points at different positions are extracted, and the ratio of the brightness of the two points and the ratio of the brightness of another two sets of points are calculated from the target object. Estimated the absolute distance of.

In the present embodiment, it is considered that the diffuse reflectance on the target object does not change when the illumination condition is changed by changing the image pickup position of the image pickup means while keeping the amount of light applied to the target object constant. Under this condition, a brightness value of an image obtained by capturing one point on the target object from two different positions and a brightness value obtained from a relative distance estimated based on the positions at which the two images are captured are It is applied to the distance estimation method in the fourth embodiment to estimate the absolute distance to the target object.

As shown in FIG. 33, using the relative distance estimated by the relative shape estimation means with reference to the image f0 and the image fi, the position from the viewpoint O of the image pickup means to the point P on the target object is detected. Find the vector. The position vector M01 from the viewpoint O to the point P on the target object, which is obtained based on the position where the image f0 is captured, is given by the following equations (10) and (11).

[Equation 66] And the position vector M11 from the viewpoint O to the point P on the target object when the image fi is used as a reference is

[Equation 67] Becomes

Further, the image f
0 and the relative distance estimated with reference to the image fi + a, the position vector M from the viewpoint O of the image capturing means to the point P on the target object obtained with the position where the image f0 is captured as a reference.
02 is from equations (10) and (11)

[Equation 68] Therefore, the position vector M22 from the viewpoint O to the point P on the target object when the image fi + a is used as a reference is

[Equation 69] Becomes

Since the position vectors M01 and M02 and the unit vectors m1 and m2 are equal, the magnification k2 and the position vector M22 are

[Equation 70] Becomes

Further, the value of the brightness of the point P estimated from each position vector from the equation (43) is as follows.

[0210]

[Equation 71] Further, assuming that the brightnesses obtained from the picked-up images f0, fi, fi + a are I0, Ii, Ii + a, respectively, the brightness ratio is given by the equation (44).

[Equation 72]

[Equation 73] Becomes

Substituting the position vectors M01, M11, M22 of the equations (66), (67) and (70) into the equations (72) and (73),

[Equation 74]

[Equation 75] Thus, two equations are obtained in which one scaling factor k1 is an unknown number. By solving these two equations, the scaling factor k1
Is required.

Further, as described in the fourth embodiment,
It is also possible to estimate the magnification from multiple points in the image. In this case, when the least squares method is applied to the ratio of luminance obtained from a plurality of points (for example, M + 1 points) on the target object and two reference images (for example, image f0 and image fi),

[Equation 76] Then, the magnification k1 that satisfies the formula (76) is obtained for a plurality of points.

Further, the magnification can be estimated from the same point on a plurality of images. In this case, if the least squares method is applied to the ratio of the luminance obtained from one point on the target object and a plurality of reference images (for example, image f0 to image fM),

[Equation 77] Then, the magnification k1 that satisfies Expression (77) is obtained for the same point on a plurality of images.

The diffuse reflectance assumed in the fourth embodiment is
Although it depends on the position of the target object, when the target object is imaged by changing the image pickup position of the image pickup means while keeping the amount of light applied to the target object constant, the diffuse reflectance on the target object does not change.

Therefore, in the present embodiment, in consideration of this point, the brightness of the image obtained by picking up one point on the target object from two different positions, and the relative estimated based on the positions at which the two images were picked up. The absolute distance to the target object can be accurately estimated by using the brightness obtained by using the specific distance. Therefore, as in the fourth embodiment, it is possible to accurately estimate the absolute shape of the target object without adding a special device or changing the structure of the device.

Next, a ninth embodiment of the present invention will be described with reference to FIG.

The ninth embodiment applies the eighth embodiment to a configuration having an endoscope having a plurality of light source irradiation ports and a light source device for alternately turning on the light emitted from each irradiation port. It is an example.

In the eighth embodiment, the illumination condition is changed by changing the image pickup position of the image pickup means, but in the present embodiment, the illumination condition is changed by alternately irradiating light from two irradiation ports. , Estimate the absolute distance to the target object.

As shown in FIG. 34, two irradiation ports A and B for irradiating illumination light are provided at the tip of the endoscope. At the same time as moving the tip of the endoscope on the target object and alternately irradiating the illumination light from the two irradiation ports A and B,
An image of the target object illuminated by each illumination light is captured and stored.

Then, the relative distance between the image pickup means and the target object is estimated from the time series image fAi (i = 0, 1, ..., N) obtained by the light emitted from the irradiation port A.

The position from the viewpoint O of the image pickup means to the point P on the target object obtained by the relative shape estimation means with reference to the position of the image pickup means when the image fA0 is picked up with reference to the image fA0 and the image fAi. The vector M0 is

[Equation 78] And the position vector Mi from the viewpoint O to the point P on the target object when the image fi is used as a reference,

[Expression 79] And

Further, the value of the brightness due to the light of the irradiation port A at the point P estimated from each position vector from the equation (43) is

[Equation 80] And the value of the brightness due to the light from the irradiation port B is

[Equation 81] Becomes

Also, the actual picked-up images fA0, fAi,
Assuming that the brightness obtained from fBi is I0, Ii, and Ii ', respectively, the ratio of the brightness can be calculated from equation (44) as follows:

(Equation 82)

[Equation 83] Becomes

Substituting the position vectors M0 and Mi of the equations (78) and (79) into the equations (82) and (83),

[Equation 84]

[Equation 85] Thus, two equations are obtained in which one scaling factor k is an unknown number. The magnification k is obtained by solving these two equations.

According to this embodiment, the absolute shape of the target object can be estimated by making a simple change to the configuration of the conventional endoscope system. Moreover, since the illumination conditions are changed by alternately irradiating light from the two irradiation ports, it is possible to accurately estimate the absolute distance without changing the image pickup position of the image pickup means.

Next, a tenth embodiment of the present invention will be described with reference to FIGS.

In the tenth embodiment, a configuration example of the position detecting means in the relative shape estimating means will be described.

The position detecting means of this embodiment includes an image capturing means for capturing images sent in time series, and a distortion aberration correcting means for correcting the distortion aberration of the image captured by the image capturing means. A power calculation means for calculating the power of the frequency characteristic of a specific area of the image captured by the image capturing means, a same point detection means for detecting the same point in the image, and a result of the same point detection means From the same point, a moving amount calculating means for calculating the moving amount of the same point, a position predicting means for predicting the position of the same point of the images captured by the moving amount calculating means, and a power comparison for comparing the power of the frequency characteristics of the respective images Means and having.

In the present embodiment, as in the first embodiment, as shown in FIG. 36A, the same object is imaged from a plurality of positions by the image pickup means, and a plurality of imaged images are picked up. Images are taken into the image processing device 3 in time series.

FIG. 35 is a flow chart showing a process of detecting the same point in a plurality of images which are time-sequentially input by the position detecting means.

By the information input device 41 such as a keyboard,
When the operator of the endoscope gives an instruction to observe the shape of a lesion or the like, the instruction is transmitted to the CPU 40 in the image processing apparatus 3 and the image processing means 47 configured in the CPU 40.
The process is started by.

First, in step S11, the initial image f0 input in time series is taken in, and in step S12, the distortion aberration is corrected. Then, in step S13, i
= 0, that is, it is determined whether the image is the initial image f0. If the image is the initial image f0, the process proceeds to step S14. If the image is any other image, the process proceeds to step S17.

At step S14, as shown in FIG. 36B, a plurality of points Pab (a = 0, 1, ..., Q, b =) are detected in order to detect the same point between images in the initial image f0.
0, 1, ..., R) are set. And step S15
Then, as shown in (c) of FIG. 36, an appropriate area S0ab including the point Pab set in step S14 is set, and FIG.
As shown in (d) of FIG.
The frequency characteristic of the partial image of 0ab is obtained, and the total power Ps0ab (Psum_0) of the appropriate frequency region (λ0 to λ1) for this partial image is calculated.

The Ps0ab obtained in step S15 is sent to step S20. Then, in step S16, i is incremented by 1, and the process returns to step S11.

In step S11, the next image f1 is fetched again, and the distortion is corrected in step S12.
The process proceeds from step S13 to step S17, and it is determined whether i = 1, that is, whether the image is the first input image. When it is determined in step S17 that the image is the first image f1, the same point is detected with respect to the plurality of points Pab set in the image f0 in step S21.

Then, in step S22, step S21
The moving amount M1 of each point between the initial image f0 and the first image f1 is calculated from the position of the same point detected in
Is sent to step S18, and in step S23, C, which is a count value indicating how many times the images with defocus have continuously occurred until the last time, is initialized to C = 0.

After that, in step S24, it is determined whether i = n, that is, whether the final image is fn. If it is not the final image, i is incremented by 1 in step S16 and the process returns to step S11.

On the other hand, in step S17, the first image f1
If it is determined that it is not, in step S18, i
From the movement amount Mi-1-c obtained in step S22 for the th image fi and the value C obtained by counting the number of consecutively defocused images generated as a result of step S25 up to the previous time, The position of the same point in the i-th image of the point Pab set in S14 is predicted.

Then, in step S19, step S1
The total power Psiab (Psum_i) of the region including the position of the same point predicted in 8 and the frequency characteristic of the region equivalent to the region S0ab set in the initial image f0 and the region equivalent to the frequency region set in the initial image f0. calculate.

Then, in step S20, step S15
The power Ps0ab obtained in step S19 is compared with the power Psiab obtained in step S19, and if the power of the i-th image is equal to or higher than the power of the initial image, it is determined that there is no defocus,
In step S21, the same point is detected for the plurality of points Pab set in the image f0. Also, step S2
If the power of the i-th image is 0 and smaller than that of the initial image, it is determined that defocusing has occurred, and step S25
C is incremented by one, i is incremented by one in step S26, and the process returns to step S11.

The above processing is repeated until the nth image fn is processed, and when it is determined in step S24 that it is the final image fn, this processing is ended.

Here, the initial image f0 and the i-th image fi
The power in the appropriate frequency range of the frequency characteristics obtained by

[Equation 86] Is required.

For the prediction of the position of the same point in the i-th image, the movement amount of the i-th image with respect to the initial image is M
Let i be the case where no focus shift has occurred up to the i-1th image.

[Equation 87] Becomes If the i−1th image is out of focus and the same point is not detected, the predicted position of the same point in the ith image is the movement amount up to the i−2nd,

[Equation 88] Can be obtained as

Also, in step S20, the initial image f
The powers in the appropriate frequency regions of the frequency characteristics obtained from the 0th and the i-th image fi are compared.

Psum_0 −α ≦ Psum_i ≦ Psum_0 + α (89) As shown in (89), the total power obtained in the i-th image fi is compared with the total power obtained in the initial image f0 with an appropriate width. Is also good.

An endoscope used for observing a living body generally observes a mucous membrane image without clear corners, contours, and textures by an optical system having a shallow depth of focus. For this reason, when those images are obtained in time series, there is a high possibility that images out of focus are included. When block matching is used for such a time-series image as in the above-mentioned Japanese Patent Laid-Open No. 6-7289, there is a possibility that accurate tracking of corresponding points cannot be performed.

On the other hand, by detecting the same point by the position detecting means as described above, in the present embodiment, the focus shift is detected for a plurality of images captured in time series, and the focus shift is detected. By detecting the same point in an image that has not occurred, it is possible to accurately track the corresponding point even in an image such as an endoscope that includes defocus. Therefore, when the imaging means is moved to capture a plurality of images of the target object in time series and the same point between the images is detected, even when a mucosal image that is out of focus is captured. It is possible to accurately detect the same point.

Next, an eleventh embodiment of the present invention will be described with reference to FIGS. 37 and 38.

The eleventh embodiment is an example in which the functional configuration of the position detecting means of the tenth embodiment is changed and a step of changing the size of the template image is added to the procedure for detecting the same point.

FIG. 37 is a flow chart showing the process of detecting the same point in a plurality of images which are time-sequentially input by the position detecting means. Steps S31 to S4 of FIG.
The processing up to the processing of the initial image f0 and the first image f1 in Step 3 is the same as steps S11 to S24 of the tenth embodiment, and a description thereof will be omitted.

When the images of the second and subsequent images fi are first captured in step S31, the distortion aberration is corrected in step S32, and the flow advances to step S38.

In step S38, the moving amount Mi-1-c obtained in step S42 for the image fi or the average moving amount calculated in step S46, which will be described later, is set in step S34 according to the result of step S40. The position of the same point in the i-th image of the point Pab is predicted. That is, when the power of the i-th image is equal to or higher than the power of the initial image in the previous step S40, the movement amount obtained in step S42 is used when the power of the i-th image is smaller than the power of the initial image in step S40. Predicts the position of the same point using the average movement amount obtained in step S46.

Then, in step S39, step S3
The sum Psum_i of the frequency characteristics of the region including the position of the same point predicted in 8 and equivalent to the region S0ab set in the initial image f0 and the power of the region equivalent to the frequency region set in the initial image f0 is calculated.

Then, in step S40, step S35
The power Psum_0 obtained in step S39 is compared with the power Psum_i obtained in step S39. If the power of the i-th image is equal to or higher than the power of the initial image, it is determined that there is no focus shift, and the image f0 is obtained in step S41. The same point is detected for the plurality of points Pab set in. If the power of the i-th image is smaller than that of the initial image in step S40, it is determined that defocusing has occurred, and step S40 is performed.
Proceed to 44.

In step S44, the size of the template image and the number of points Pab set in step S34 are reduced, and in step S45, the same point is detected with respect to the set template image size and points in the image. . For example, increase the size of the template image and reduce the number of points in the image by half.

In step S46, the moving amount is calculated from the position of the same point detected in step S45 as the average moving amount between images in which defocuses have occurred continuously until the present,
The average value of the movement amount of each point is calculated.

Thereafter, i is incremented by 1 in step S47 and the process returns to step S31.

The above processing is repeated to process the nth image fn, and when it is determined in step S43 that the image is the final image fn, this processing ends.

As the observation point moves away from the focal length, the power of the obtained image in the high frequency region becomes smaller. For this reason, in template matching, as the power in the high frequency region becomes smaller, the accuracy of detecting the same point between a plurality of images becomes worse.

Therefore, in the present embodiment, the size of the template image is changed according to the power of the frequency characteristic to prevent the accuracy of detecting the same point from being deteriorated.

FIG. 38 is a diagram showing the relationship between the amount of defocus and the size of the template image. Here, the ratio Pratio between the power Psum_0 of the initial image obtained in step S35 and the power Psum_i of the i-th image obtained in step S39 is used as an amount representing the focus shift.
Is used.

[0261]

[Equation 90] The size Ts0 of the template image used when the power of the frequency characteristic is calculated in step S35 is used up to the defocus of a specific size. If the focus shift becomes larger than this, the size of the template image is set to Tsi so as to increase linearly as Pratio increases in accordance with the ratio Pratio between the power Psum_0 of the initial image and the power Psum_i of the i-th image. change.

[0262]

Tsi = Pratio * Ts0 (91) Alternatively, the size of the template image may be determined by giving an appropriate weight w to the power ratio.

[0263]

Tsi = w * Pratio * Ts0 (92) Further, similar to the tenth embodiment, the power in the appropriate frequency domain of the frequency characteristics obtained in the initial image f0 and the i-th image fi in step S40. I compared each

(93) Psum_0−α ≦ Psum_i ≦ Psum_0 + α (93) As shown in (93), the total power obtained in the i-th image fi is compared with the total power obtained in the initial image f0 with an appropriate width. Is also good.

As described above, by detecting the same point by template matching in the position detecting means,
In this embodiment, a focus shift is detected for a plurality of images captured in time series, and if the focus shift is large, the template image size is changed to perform template matching, and an average movement amount is calculated. As a result, it is possible to reliably predict the position of the same point even for an image such as an endoscope that includes defocus.

Further, in the tenth embodiment, if a focus shift occurs in a plurality of images or the movement of the tip of the endoscope is large, an error is likely to occur in the prediction of the position of the same point. However, according to the present embodiment, it is possible to predict the position of the same point from the image in which the defocus occurs, so that it is possible to prevent the inconvenience as described above, and even when the defocus occurs, the position of the same point is always accurate. It becomes possible to perform detection.

[Appendix] (1) Relative shape of the target object from an endoscope having an image pickup means for picking up the same target object from different positions and a plurality of images picked up by the image pickup means , A distance estimation means for estimating a distance from the image pickup means to an appropriate position on the target object, a relative shape of the target object estimated by the relative shape estimation means, and the distance estimation An endoscopic image processing device, comprising: an absolute shape estimating means for estimating an absolute shape of the target object from the distance to the target object estimated by the means.

(2) The distance estimating means calculates the magnitude of defocus at the same point in the plurality of images, and the minimum for detecting the point where the calculated defocus is minimum. Note 1. The point detection means is included, and the distance to the target object is estimated by setting the distance from the target object at the detected minimum point to the imaging means to the focal length of the imaging means. The endoscopic image processing device according to item 1.

(3) The distance estimating means includes halation detecting means for detecting the position of halation occurring in the image, and normal vector estimating means for estimating a normal vector at the halation position on the target object. An emission position vector estimation means for estimating a vector from the viewpoint of the image pickup means to the emission position of the light source, and a halation position vector estimation means for estimating a vector from the viewpoint of the image pickup means to the halation position on the target object are provided. The endoscopic image processing according to Appendix 1, wherein the distance to the target object is estimated by estimating the distance from the viewpoint to the halation position on the target object based on the relationship between the estimated three vectors. apparatus.

In Supplementary Notes 1 to 3, deviations and halations from the focal length are detected from time-series images obtained by the image pickup means, and the detected features are used to detect an appropriate point on the target object from the image pickup means. By estimating the distance to the position, the absolute distance to the target object is accurately estimated.

(4) Laser light generating means for generating laser light, and light guiding means for guiding the laser light from the laser light generating means to the tip of the endoscope in which the image pickup means is provided,
The endoscopic image processing apparatus according to appendix 1, further comprising:

(5) The distance estimation means estimates an emission position vector estimation means for estimating a vector from the viewpoint of the image pickup means to the emission position of the laser beam, and the laser irradiated on the target object from the viewpoint of the image pickup means. Irradiation position vector estimating means for estimating a vector up to the irradiation position of light, and a laser irradiation vector for estimating a laser irradiation vector from the emission position of the laser light to the position irradiated on the target object from the estimated two vectors. Since the outer product of the two laser irradiation vectors obtained from different positions of the estimation means and the image pickup means becomes 0, the absolute distance for estimating the distance from the image pickup means to the position on the target object where the laser light is emitted. The endoscopic image processing device according to attachment 4, further comprising: an estimating unit.

In Supplementary Notes 4 and 5, laser light is irradiated from the tip of the endoscope, a plurality of images of the target object irradiated with the laser light are picked up, and the images from the image pickup means to an appropriate point position on the target object. By estimating the distance, the absolute distance to the target object is accurately estimated.

(6) The distance estimating means obtains a luminance ratio between two appropriate points on the image obtained by the image pickup means, and this luminance ratio is calculated between the viewpoint of the image pickup means, the light source and the target object. Expressed by an equation having a magnification that is derived by a relative positional relationship as a variable, a plurality of equations relating to the magnification are derived by combining a plurality of points on the target object, and the magnification is estimated by solving the plurality of equations. The endoscopic image processing apparatus according to appendix 1, further comprising a magnification estimating unit, which estimates a distance to the target object from the magnification.

(7) The magnification estimating means divides a plurality of equations calculated by a combination of a plurality of points on the target object into groups of the number necessary for obtaining the magnification, and obtains each group. 7. The endoscopic image processing apparatus according to appendix 6, comprising means for estimating one magnification by averaging the magnifications.

(8) The magnification estimating means determines the brightness ratio between two appropriate points on the image obtained by the image pickup means and the relative positional relationship between the viewpoint of the image pickup means, the light source and the target object. 7. The endoscopic image processing apparatus according to attachment 6, comprising means for estimating a magnification that minimizes the sum of squares of the difference between the ratio of the brightness with the derived magnification as a variable.

(9) The distance estimation means includes magnification estimation means for estimating magnification using a plurality of two points of the same color on the image obtained by the image pickup means. Mirror image processing device.

(10) The magnification estimating means calculates the ratio of luminance between two points of the same color on the image obtained by the image pickup means, and the relative position of the viewpoint of the image pickup means, the light source and the target object. Appendix 6 comprising means for estimating the magnification such that the sum of squares of the difference between the luminance ratio with the magnification derived from the relationship as a variable and the weighting function representing the color identity is minimized. Endoscopic image processing device.

(11) The distance estimating means has a magnification estimating means for estimating the magnification using a plurality of sets of points on the image obtained by the imaging means and the surroundings of the correlation value having a sharp shape. The endoscopic image processing device according to appendix 1.

(12) The endoscopic image processing device according to attachment 11, wherein the correlation value used in the distance estimating means is a value of a cross correlation.

(13) The endoscopic image processing apparatus according to attachment 11, wherein the correlation value used in the distance estimating means is a value of autocorrelation.

(14) The magnification estimating means is configured to measure the ratio of luminance between two points having a correlation value on the image obtained by the image pickup means larger than an appropriate threshold value, a viewpoint of the image pickup means, a light source and a target object. From the means of estimating the magnification such that the sum of the squares of the difference between the luminance with the magnification derived from the relative positional relationship with and the weighting function obtained from the correlation value becomes the minimum. The endoscopic image processing device according to appendix 6.

(15) The distance estimating means determines the normalized brightness of an appropriate point on the image obtained by the image pickup means and the relative positional relationship between the viewpoint of the image pickup means, the light source and the target object. It has a magnification estimating means for estimating the magnification such that the sum of squares of the difference between the normalized luminance with the derived magnification as a variable and the difference between the luminance and the target object is obtained from the estimated magnification. The endoscopic image processing device according to attachment 1, which estimates the distance between the two.

(16) The endoscopic image processing device according to attachment 15, wherein the normalized luminance used in the distance estimating means is normalized by the average value of the luminance around the points on the image.

(17) The endoscopic image processing apparatus according to attachment 6, wherein the ratio of the luminance between two points used in the distance estimating means is obtained from the same point imaged from different positions of the image capturing means.

(18) The image pickup means has a plurality of irradiation openings for irradiating illumination light, and the distance estimation means has the same number of two images obtained according to the light emitted from the irradiation openings. The ratio of luminance between points is expressed by an equation with a variable that is a magnification that is derived by the relative positional relationship between the viewpoint of the image pickup unit, the position of the irradiation opening, and the target object, and a combination of a plurality of points on the target object. A plurality of equations relating to the magnification are derived by, and a magnification estimating unit for estimating the magnification by solving the plurality of equations is provided, and the distance from the imaging unit to the target object is estimated from the estimated magnification. The endoscopic image processing apparatus according to attachment 1.

In Supplementary Notes 6 to 18, the image pickup means is calculated from the ratio of the luminance estimated from the relative positional relationship between the viewpoint of the image pickup means, the light source and the target object and the luminance ratio of the image actually obtained by the image pickup means. By estimating the distance from to the position of an appropriate point on the target object, the absolute distance to the target object can be accurately estimated.

(19) The relative shape estimation means detects the position of the same point on the image from the plurality of images picked up by the image pickup means, and the same position detection means detects the same position. Imaging position estimation means for estimating the position of each imaging means with respect to the position of the target object from the position of the point,
The endoscopic image processing apparatus according to appendix 1, further comprising: a relative distance estimating unit that estimates a relative distance from the image capturing unit to a target object based on a result of the position detecting unit and the image capturing position estimating unit. .

(20) The position detecting means detects the position of the same point on the image, and the characteristic detecting means for detecting the characteristics of the partial image around the position predicted by the position predicting means. 20. The endoscopic image processing apparatus according to appendix 19, further comprising: a unit and a same point detection unit that performs a process of detecting the same position on the image according to the detected characteristic.

(21) The endoscopic image processing apparatus according to attachment 20, wherein the position predicting means predicts from a position estimated from the movement amount of the position of the same point up to the previous time in the images obtained in time series. .

(22) The endoscopic image processing apparatus according to attachment 20, wherein the feature detecting means detects a feature from the frequency characteristic of the partial image.

(23) The same point detecting means detects the same point position detecting means for detecting the position of the same point on the image, and the movement amount of the same point position from the result of the same point position detecting means. 21. The endoscopic image processing device according to attachment 20, comprising: a moving amount calculating unit for calculating.

(24) The endoscopic image processing apparatus according to attachment 23, wherein the same-point position detection means uses template matching.

(25) The same-point detecting means has a counting means for counting the number of times that the position of the same point is not continuously detected in the image to be processed. Mirror image processing device.

In Supplementary Notes 19 to 25, an image having a defocus is detected from the images obtained in time series, the image having the defocus is not used, and the same images are obtained. By predicting the position of the point from the amount of movement up to the previous time and detecting the position of the same point, the same point can be accurately detected even when an image out of focus is captured.

(26) The same point detecting means includes template changing means for changing the number of points for detecting the positions of the same points on the image and the size of the template image, and the number of changed detected points and the template. The same point position detection means for detecting the position of the same point on the image from the image, and the movement amount calculation means for calculating the average movement amount of the position of the same point from the result of the same point position detection means 21. The endoscopic image processing device according to attachment 20, comprising:

(27) The template changing means is
27. The endoscopic image processing device according to attachment 26, wherein the number of detection points and the size of the template are determined according to the features of the partial images detected by the feature detection means.

In Supplementary Notes 26 and 27, the image in which the defocus is generated is detected from the images obtained in time series,
The size of the template image is changed between images in which defocus occurs, and the position of the same point on the image is detected. At this time, based on the detected position of the same point, the average amount of movement between images in which defocus is generated is calculated, and the position of the same point on the image is predicted from the amount of movement to predict the position of the same point. By detecting the position, the same point can be accurately detected even when an out-of-focus image is captured.

[0298]

As described above, according to the present invention, the absolute distance from the image pickup means to the target object can be accurately and easily based on the images taken from different positions for the same target object. There is an effect that can be estimated.

[Brief description of drawings]

FIG. 1 is an overall configuration diagram of an electronic endoscope system according to a first embodiment.

FIG. 2 is a block diagram showing a block configuration of an electronic endoscope apparatus.

FIG. 3 is a block diagram showing the configuration of an image processing apparatus.

FIG. 4 is a block diagram showing a functional configuration of an image processing unit including a CPU.

FIG. 5 is a block diagram showing a configuration of an image processing unit configured by hardware.

FIG. 6 is a block diagram showing the configuration of a recording device.

FIG. 7 is a block diagram showing a modified example of the input unit of the image processing apparatus.

FIG. 8 is a block diagram showing a functional configuration of relative shape estimation means.

FIG. 9 is a block diagram showing a functional configuration of distance estimation means.

FIG. 10 is a flowchart showing the overall flow of processing for estimating the relative shape of an object.

11 is an operation explanatory view for explaining each processing of FIG.

FIG. 12 is an explanatory diagram showing a state in which a shift map is created from image data obtained in time series.

FIG. 13 is an explanatory diagram showing a coordinate system of central projection.

FIG. 14 is an explanatory diagram showing a relationship between a rotation vector and a translation vector related to the motion of the image pickup means.

FIG. 15 is an explanatory diagram showing a change in distance to a target object due to movement of the image pickup means.

FIG. 16 is an explanatory diagram showing how to obtain the frequency characteristics of each image obtained in time series.

FIG. 17 is a characteristic diagram showing frequency characteristics of image data.

FIG. 18 is an explanatory diagram showing a positional relationship between a light source and an imaging unit and a target object when halation occurs.

FIG. 19 is an explanatory view showing a normal vector of a tangent plane at a point P on an object and a position coordinate relation of a viewpoint O and a light source S.

FIG. 20 is an operation explanatory view showing an example of a method of estimating a normal vector of a tangent plane at a point P on an object.

FIG. 21 is a structural explanatory view showing the overall structure of an endoscope system according to a third embodiment.

FIG. 22 is an operation explanatory view illustrating an operation of estimating a distance to a laser light irradiation position on a target object.

FIG. 23 is an explanatory diagram showing the relationship between the positions of the light source and the image pickup means and the brightness value of a point on the target object obtained by the image pickup means.

FIG. 24 is an explanatory diagram showing a relationship between a vector from a light source to an appropriate point on the target object and a normal vector of the target object at one point.

FIG. 25 shows two points A on the target object from the viewpoint O of the imaging means,
Explanatory diagram showing vectors up to B

FIG. 26 is an operation explanatory view showing an example of a method of estimating a light source position vector.

FIG. 27 is an operation explanatory view showing an example of a method of estimating a light source position vector.

FIG. 28 is an explanatory diagram showing a template image set as a specific area in the image.

FIG. 29 is a characteristic diagram showing the value of the cross-correlation for each point in the image.

FIG. 30 is a characteristic diagram showing changes in cross-correlation values according to patterns in an image.

FIG. 31 is an operation explanatory view showing the procedure for calculating the value of the cross-correlation regarding the template image.

FIG. 32 is an operation explanatory view showing a procedure of detecting a point having a sharp cross-correlation value shape.

FIG. 33 is an explanatory diagram showing a position vector from a viewpoint O of each imaging means to a point P on a target object in a plurality of images captured by moving the imaging means.

FIG. 34 is an operation explanatory view in the case where two irradiation ports A and B are provided at the distal end portion of the endoscope and a plurality of images are picked up by light from each irradiation port.

FIG. 35 is a flowchart showing a process of detecting the same point in a plurality of images input in time series by the position detecting means according to the tenth embodiment.

FIG. 36 is an operation explanatory view showing a procedure of calculating power in an appropriate frequency region of frequency characteristics of each image for a plurality of images obtained in time series.

FIG. 37 is a flowchart showing a process of detecting the same point in a plurality of images input in time series by the position detecting means according to the eleventh embodiment.

FIG. 38 is an explanatory diagram showing the relationship between the amount of defocus and the size of the template image.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Endoscope system 2 ... Electronic endoscope apparatus 3 ... Image processing apparatus 4 ... Recording apparatus 5, 8 ... Monitor 6 ... Electronic endoscope 7 ... Observing apparatus 7A ... Light source section 7B ... Signal processing section 16 ... Tip section Reference numeral 19 ... Imaging unit 40 ... CPU 42 ... Main storage device (RAM) 45 ... ROM 47 ... Image processing means 50, 62 ... Relative shape estimation means 51, 63 ... Distance estimation means 52, 64 ... Absolute shape estimation means

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Tetsuo Nonami 2-43-2 Hatagaya, Shibuya-ku, Tokyo Inside Olympus Optical Co., Ltd.

Claims (1)

[Claims] 1. An endoscope having image pickup means for picking up the same target object from different positions, and a relative for estimating a relative shape of the target object from a plurality of images picked up by the image pickup means. Shape estimating means, distance estimating means for estimating a distance from the imaging means to an appropriate position on the target object, relative shape of the target object estimated by the relative shape estimating means, and the distance estimating means And an absolute shape estimating means for estimating an absolute shape of the target object from the distance to the target object.