JPH0870473A - Communication method of stereoscopic image using distance information - Google Patents

Abstract

(57) [Abstract] [Purpose] To provide a communication method for stereoscopic images using distance information. It also provides a stereoscopic image communication system that enables various types of processing of reproduced image information. [Structure] In a stereoscopic image communication method including creation, recording, transmission, and reproduction processing of stereoscopic image information in a stereoscopic image service provided via a communication network, a distance as subject image information is measured in pixel units, The subject image information newly added as distance information is transmitted.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereoscopic image communication system, and more particularly to inputting / recording / transmitting and reproducing subject information in a stereoscopic image service provided through a communication network.

[0002]

2. Description of the Related Art Conventionally, regarding image capturing / recording / transmission / playback technology, a case where communication is assumed will be described.
FIG. 6 shows the flow of the conventional image information imaging / recording / transmission / reproduction technology. In FIG. 6, the transmission side and the reception side are connected via a communication network. 101 is a photoelectric conversion unit,
The subject image information is optically processed at 101 to decompose the image information into pixel information. Information about brightness and color is added to each pixel. A transmission scan processing unit 102 scans an image and converts it into an image signal. A transmission processing unit 103 performs high-efficiency digital encoding / modulation / demodulation processing of an image signal as transmission processing. A reception scanning processing unit 104 performs decoding reception scanning of the received image code. An electro-optical conversion processing unit 105 reproduces a subject image from pixel information by optical processing.

FIG. 7 is a diagram showing an image communication process from image pickup to reproduction according to a conventional technique. In FIG. 7, 20
An image pickup processing unit 1 converts subject information into luminance / color signals. A compression processing unit 202 compresses the image information if necessary. 203 is a recording unit,
If necessary, record the image information. Reference numeral 204 is an encoding processing unit, and 205 is a transmission processing unit. Reference numeral 206 is a decoding processing unit, and 207 is a recording unit on the receiving side, which records if necessary. A decompression processing unit 208 includes a compression processing unit 202.
The image information compressed by is restored. An image processing unit 209 restores the brightness / hue information. Reference numeral 210 denotes a projection image generation processing unit, which reproduces the imaging information. 21
Reference numeral 1 is a projection unit including a screen and the like.

[0004]

In the above-mentioned conventional image communication system, the distance information of the object from the camera is ignored and only the plane image obtained by projecting the three-dimensional space onto the two-dimensional plane is transmitted. For example, in the case of a television system, a flat image projected on a photocathode by an optical system of a camera is a transmission target. If a human being is the recipient, not all wavelength components need to be sent due to their visual characteristics. Only the signals corresponding to red, green and blue need to be transmitted according to the principle of the three primary colors. Specifically, it is sufficient to transmit only three types of images obtained by using appropriate weighting functions wR (λ), wG (λ), and wB (λ) having the wavelength λ as a variable. As described above, in the conventional technology, there is no input of a distance parameter that expresses the perspective of an image. Therefore, the perspective of subject information is reproduced and displayed only by brightness and hue, and there is no method for transmitting distance information. It was

The conventional planar image information, which lacks the distance information, has a large limitation when using various image communication services. For example, these operations such as changing the observation angle of the subject, extracting coordinate data of the subject, and compressing redundant information of image pickup information by adjacent cameras are important in stereoscopic image communication. However, it is difficult to perform the above-described image processing from the conventional planar image information in which the distance information is missing.

An object of the present invention is to provide a communication method for stereoscopic images using distance information.

Another object of the present invention is to provide a stereoscopic image communication system which enables various processing of reproduced image information.

The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

[0009]

The outline of the representative one of the inventions disclosed in the present application will be briefly described as follows.

(1) Creation, recording, and transmission of stereoscopic image information in a stereoscopic image service provided via a communication network
In a stereoscopic image communication method including a reproduction process, subject image information to which distance information obtained by measuring a distance as subject image information in pixel units is newly added is transmitted.

(2) The subject image information to which the distance information is newly added is picked up.

(3) An image is reproduced by using the subject image information to which the distance information is newly added.

(4) Subject image information to which the distance information is newly added is collectively compressed / decompressed.

(5) The subject image information newly added with the distance information is collectively encoded / decoded.

(6) The image is processed and reproduced using the subject image information to which the distance information is newly added.

That is, with respect to all the subject information in the viewfinder, the distance is detected by using an optical system distance measuring device such as infrared ray, laser, and ultrasonic wave. This is a stereoscopic image communication method in which the detected distance information is transmitted together with the image information, and various image processing is possible when processing on the reproducing side as a stereoscopic image.

[0017]

According to the above-described means, since the distance information between the subject and the camera is added to the image information, the distance information of the captured subject is recorded and transmitted, and the distance information is recorded at the receiving side together with the distance information. By making the best use of the reproduction, the receiving side can reproduce the pseudo three-dimensional image and realize the communication of the three-dimensional image that enables various processing of the reproduced image information.

[0018]

Embodiments of the present invention will be described below in detail with reference to the drawings.

In all the drawings for explaining the embodiments, parts having the same function are designated by the same reference numerals, and the repeated description thereof will be omitted.

FIG. 1 is a diagram showing an operation processing image of input / recording / transmission / recording / processing / reproducing of a stereoscopic image communication system according to an embodiment of the present invention, and 301 is an image pickup processing unit, 3
Reference numeral 02 is a distance distribution processing unit. A communication output processing unit 303 performs communication output processing on the luminance / hue image information output by the imaging processing unit 301 and the distance information output by the distance distribution processing unit 302. A compression processing unit 304 compresses the input subject information, if necessary. 30
A recording unit 5 writes pixel information and distance information of subject information, if necessary. 306 is an encoding processing unit,
Reference numeral 307 is a transmission processing unit. The transmission processing unit 307 performs transmission processing to the communication network.

Decoding processor 308 on the receiving side decodes a received signal. A recording unit 309 on the receiving side reads out subject information if necessary. 310
Is a decompression processing unit. When the subject information read from the recording unit 309 has been compressed, it is decompressed and delivered to each processing unit.

Reference numeral 311 is a distance processing unit, 312 is an image processing unit, and 313 is a reproduced image processing information input unit. Here, the subject information is detected by the imaging information input unit of the method of the present embodiment, and various image processing processes are performed from the original image information by using the added distance information. For example, it is possible to generate subject video information from a new viewpoint.

Reference numeral 314 denotes a projection image generation unit, which executes processing for the subject information processing results of the distance processing unit 311 and the image processing unit 312, and the image information processing command by the reproduced image processing information input unit 313. That is, the projection image generation unit 314 comprehensively processes the processing instruction processing by the distance processing unit 311, the image processing unit 312, and the reproduction image processing information input unit 313.

Reference numeral 315 is a projection unit, which is an output unit for the image that has been comprehensively processed by the projection image generation unit 314.

FIG. 2 is a flow chart showing an operation processing procedure according to the communication system of this embodiment, and 401 is pixel information.
In each pixel information, the signal-processed luminance, color, and distance are recorded as subject information. As a general rule, this pixel information is sent to the receiving side, but as a variation of the system, if the designer of the communication system of the present invention requires, each processing of compression / decompression / recording / processing is processed as an inserting section. It is possible to install a request node.

In the processing according to the communication method of this embodiment, as shown in FIG. 2, it is judged in step 402 whether or not the compression processing is desired, and if the compression processing is desired (YES),
In step 402 *, the compression process is performed and the process proceeds to step 403. If the compression process is not desired (NO), the compression process is not performed immediately and the process proceeds to the next step 403.

In step 403, it is determined whether the recording process is desired to be written. If the recording process is desired to be written (YES), the writing process is performed in step 403 * and the process proceeds to step 404. If the writing is not desired (NO), the writing process is not performed and the process immediately proceeds to the next step 404. In step 404, the pixel information 401 or the compressed pixel information is transmitted to the receiving side.

In step 405, the receiving side determines whether or not the reading of the recording process is desired. If the recording process is desired (YES), the reading process is performed in step 405 *, and the process proceeds to step 406 to record. If the reading is not desired (NO), the process of recording and reading is not performed, and the process immediately proceeds to the next step 406.

In step 406, it is determined whether or not the decompression process is desired. If the decompression process is desired (YES), the decompression process is performed in step 406 * and the process proceeds to step 407. (NO), the process directly proceeds to the next step 407 without performing the decompression process.

In step 407, it is determined whether the image processing is desired, and if the image processing is desired (YE
S), the image processing is performed in step 407 * and the process proceeds to step 408. If the image processing is not desired (N)
O), the process immediately proceeds to the next step 408 without performing the image processing. In step 408, the processing result of step 407 is reproduced and output.

Next, the three most characteristic processing units in the communication system of this embodiment, that is, the imaging information input unit, the compression processing unit, and the processing unit will be described in detail.

(A) Image pickup information input section In the operation processing of the image pickup information input section of the present embodiment, the distance measurement value to the subject is signal-processed for each pixel and recorded together with the luminance / color signal which is the conventional pixel information. Use a camera that does.

FIG. 3 is a flow chart showing the operation processing procedure of the imaging information input section of this embodiment. In FIG. 3, reference numeral 500 is a lens unit, 501 is an image pickup processing unit (corresponding to 301 in FIG. 1), and 502 and 503 are luminance signals and color signals, respectively, which are obtained by using the conventional image pickup information input technology. 51
1 is a distance measurement sensor, 512 is a distance distribution processing unit (corresponding to 302 in FIG. 1), 513 is distance information, and 520 is a communication output processing unit (corresponding to 303 in FIG. 1).

Next, the process of distance signal processing in the main image pickup information input section will be described with reference to FIG. Simultaneously with the imaging of the subject, the distance measuring sensor 511 measures the distance between the target object and the distance measuring sensor 511. The measured distance information is quantized by the distance distribution processing unit 512,
Signal processed.

That is, the distance distribution processing unit 512 obtains new pixel information by utilizing pixel division of the image pickup information, and the distance information 513 in which the new pixel information is signaled is recorded for each corresponding pixel. In each pixel, each information of brightness, hue and distance is recorded, and the communication output processing unit 520 processes the subject information recorded in each pixel into communicable information.

(B) Compression processing unit The enormous amount of information in a stereoscopic image poses a problem. The subject image information increases in proportion to the number of cameras and spatial sampling when capturing the subject. However, taking the case of a simple twin-lens camera as an example, the imaging information in this case is only the displacement of the horizontal position of the camera, and the correlation between the two imagings is extremely high, and therefore the high imaging Efficient compression is possible. In the image pickup unit of the method of this embodiment, since the distance information between the camera and the object is recorded in the pixel as the object information, the correlation of the pixel information is reduced by the method of using the distance information and the reduction of the redundant information and the difference information. Is added as header information to the image pickup pixel information from one direction. By this process, information from two directions can be compressed into one direction + difference information. Furthermore, this system can be applied to a compression system for adjacent N (N is an integer) eye camera imaging information.

(C) Processing unit This is a process for processing the received image information in response to a request from the receiving side. Specifically, the following processing can be performed.

(C-1) Replacement of background According to the method of this embodiment, it is possible to reproduce a picked-up image using distance. FIG. 4 is a relational diagram of distance information representing an image extraction situation based on the distance and spatial sampling.
In FIG. 4, d1, d2, d
Sampling in 3 stages of 3 is performed. Image information can be managed for each distance. Therefore, this image information
When the image pickup information of the distance information is reproduced, for example, the image information of the distance d3 can be replaced with another image information.

(C-2) Coordinate data conversion Each pixel is provided with distance information as image information, which information can be converted into a three-dimensional coordinate representation based on the following principle.

FIG. 5 is a diagram showing the conversion from the distance distribution diagram based on the pixel information into a three-dimensional coordinate system, where P is one feature point of the subject. The distance information at P is recorded in the pixel including the point P at the same time as the image information.

Introduces an x, y coordinate system whose origin is the center of a rectangle in the finder. At this time, the coordinate of P is (X
0, Y0). The camera installation point is in the direction perpendicular to the origin O of this xy plane.

Let z be a straight line connecting the origin and the camera installation point C0. The coordinates (x, y, z) of the feature point P of the subject in this x, y, z coordinate system are obtained. x is X0 and y is Y0. On the other hand, the camera installation point C0 is on the z-axis. The distance d recorded in the pixel P is also given. That is, the distance between C0 and P is d.

As can be seen from the above, the z coordinate of P can be obtained. Therefore,
By performing the process for each feature point, the subject information can be recorded as coordinate data. By this method, the feature point coordinate data of an arbitrary subject can be used to generate an important database in CG.
On the other hand, using this coordinate system, it is possible to reproduce an image when the line of sight is changed. This is because by inputting new camera set point information from the reproduced image processing information input section using this coordinate expression, distance information at all subject generation points at the new viewpoint is generated, and image information at the new viewpoint is obtained. Can be created.

As can be seen from the above description, according to the present embodiment, the stereoscopic image information is transmitted by using the method of adding the object-camera distance information measured on the imaging side as new object image information. It is possible to perform image processing such as a compression processing method and line-of-sight change for.
In other words, by creating a subject database, processing in CG, and high-rate compression of N-eye image information, stereoscopic image communication is made a realizable area as a communication service, so in an office where telecommuting is introduced,
It is possible to provide a nuance of the image, the atmosphere of the meeting, the line of sight, etc. by connecting the remote discrete environments by communication.

Although the present invention has been specifically described based on the above-described embodiments, the present invention is not limited to the above-mentioned embodiments and can be variously modified without departing from the scope of the invention. Is.

[0046]

The effects obtained by the representative one of the inventions disclosed in the present application will be briefly described as follows.

(1) A stereoscopic image communication system using distance information can be realized. That is, with respect to all the subject information in the viewfinder, it is possible to detect the distance using an optical distance measuring device such as infrared ray, laser, ultrasonic wave, etc., and transmit the detected distance information together with the image information.

(2) Since the distance information between the subject and the camera is added to the image information, the captured distance information of the subject is recorded and transmitted, and the reception side reproduces the distance information together with the distance information. By doing so, communication of a stereoscopic image that enables reproduction of a pseudo stereoscopic image on the receiving side and various processing of reproduced image information is realized.

(3) Since stereoscopic image communication is set as a feasible area for communication services, in an office where telecommuting is introduced, remote discrete environments are connected by communication, and the atmosphere and line of sight of images and meetings Can provide nuances and the like.

[Brief description of drawings]

FIG. 1 is a diagram showing an operation processing image of input / recording / transmission / recording / processing / playback in a stereoscopic image communication system according to an embodiment of the present invention.

FIG. 2 is a flowchart showing an operation processing procedure according to the communication system of the present embodiment.

FIG. 3 is a flowchart showing an operation processing procedure of an image pickup information input unit of the present embodiment.

FIG. 4 is a relationship diagram of distance information and spatial sampling representing an image extraction situation based on distance according to the present embodiment.

FIG. 5 is a graph 3 based on a distance distribution map based on pixel information according to the present embodiment.
It is a figure showing the conversion to a three-dimensional coordinate system.

FIG. 6 is a diagram for explaining a conventional image communication mode. Is.

FIG. 7 is a diagram showing a series of operation processing images from imaging to reproduction in conventional image communication.

[Explanation of symbols]

301 ... Imaging processing unit, 302 ... Distance distribution processing unit, 303
... communication output processing unit, 304 ... compression processing unit, 305 ... recording unit, 306 ... encoding processing unit, 307 ... transmission processing unit, 30
8 ... Receiving side decoding processing unit, 309 ... Receiving side recording unit,
Reference numeral 310 ... Decompression processing unit, 311 ... Distance processing unit, 312 ... Image processing unit, 313 ... Reproduced image processing information input unit, 314 ...
Projection image generation unit, 500 ... Lens unit, 501 ... Imaging processing unit, 511 ... Distance measurement sensor, 512 ... Distance distribution processing unit, 520 ... Communication output processing unit.

Claims (6)

[Claims] 1. In a stereoscopic image communication method including creation, recording, transmission, and reproduction processing of stereoscopic image information in a stereoscopic image service provided via a communication network, a distance as object image information is measured in pixel units. Then, the stereoscopic image communication method is characterized in that the subject image information newly added as the distance information is transmitted. 2. The stereoscopic image communication system according to claim 1, wherein the subject image information to which the distance information is newly added is imaged. 3. The stereoscopic image communication system according to claim 1, wherein the image is reproduced by using the subject image information to which the distance information is newly added. 4. The stereoscopic image communication system according to claim 1, wherein the subject image information to which the distance information is newly added is collectively compressed / decompressed. 5. The stereoscopic image communication system according to claim 1, wherein the subject image information newly added with the distance information is collectively encoded / decoded. 6. The stereoscopic image communication system according to claim 1, wherein the image is processed and reproduced by using the subject image information to which the distance information is newly added.