DE19906830A1 - Method and arrangement for storing at least one image by a computer - Google Patents

Abstract

The invention relates to a method for storing at least one image by means of a computing unit, according to which relational information is stored such that it is associated with said at least one image.

Description

The invention relates to a method and an arrangement for Storage of at least one image by a computer. A method for image compression with associated Arrangement is known from [1]. The known method serves in the MPEG standard as coding method and based in essentially on the hybrid DCT (Discrete Cosine Transformation) with motion compensation. A similar The method is used for video telephony with n × 64 kbit / s (CCITT Recommendation H. 261), for the TV contribution (CCR recommendation 723) with 34 or 45 Mbit / s and for multimedia applications at 1.2 Mbit / s (ISO-MPEG-1). The hybrid DCT exists from a temporal processing level that the Relationships of successive images exploits, and a local processing level that the Exploits correlation within an image.

The local processing (intraframe coding) corresponds essentially the classic DCT coding. The picture will broken down into blocks of 8 × 8 pixels, each using DCT can be transformed into the frequency domain. The result is a matrix of 8 × 8 coefficients that approximate the two-dimensional spatial frequencies in the transformed image block reflect. A coefficient with frequency 0 (DC component) represents an average gray value of the image block.

After the transformation, data expansion takes place. However, in natural picture templates there is a Concentration of energy around the DC component (DC value) take place while the highest frequency coefficients mostly Are zero.  

The next step is spectral weighting of the coefficients so that the amplitude accuracy of the high-frequency coefficient is reduced. This is useful the properties of the human eye, the high Resolves spatial frequencies less accurately than low ones.

A second step of data reduction takes the form of a adaptive quantization by which the amplitude accuracy the coefficient is further reduced or by which small amplitudes are set to zero. The measure of Quantization depends on the fill level of the output buffer ab: If the buffer is empty, fine quantization takes place, see above that more data is generated while at full buffer is coarser quantized, which increases the amount of data reduced.

After quantization, the block is scanned diagonally ("zigzag" scanning), followed by entropy coding, which brings about a further reduction in data. Two effects are used for this:

• 1. The statistics of the amplitude values (high amplitude values occur less frequently than small ones, so that the rare Long events and short common events Code words are assigned (variable length coding, VLC). This results in an average lower data rate than with a coding with fixed Word length. The variable rate of the VLC is then smoothed in the buffer.
• 2. One takes advantage of the fact that from a certain In most cases, the value only follows zeros. Instead of all these zeros, only one is transmitted EOB code (End Of Block), resulting in a significant Coding gain in the compression of the image data leads. Instead of the output rate of 512 bits, for example, are then only  46 bit for this block to transmit, which one Compression factor of over 11 corresponds.

Another compression gain is obtained through the temporal processing (interframe coding). For coding Difference images require less data rate than for the original pictures, because the amplitude values are far less.

However, the time differences are only small, though the movements in the picture are also slight. However, they are Large movements in the picture, so there are big differences that are again difficult to code. For this reason, the Image-to-image motion measured (motion estimation) and before the difference is compensated (motion compensation). The movement information is combined with the image information transmitted, usually only one motion vector per Macroblock (e.g. four 8 × 8 picture blocks) is used.

Even smaller amplitude values of the difference images will be received if instead of the prediction used motion-compensated bidirectional prediction is used.

With a motion-compensated hybrid encoder, that won't be the case Image signal itself transformed, but the temporal Differential signal. For this reason, the coder also has a temporal recursion loop, because the predictor must Prediction value from the values of those already transmitted Calculate (encoded) images. An identical time Recursion loop is in the decoder, so that coder and Decoders are fully synchronized.

There are mainly three in the MPEG-2 coding method different methods with which images are processed can:  

I-pictures: With the I-pictures there is no temporal prediction used, d. that is, the image values become direct transformed and encoded. I pictures used the decoding process without knowing the to be able to start again in the past or to resynchronize To achieve transmission errors.

P-pictures: Using the P-pictures, a temporal prediction is made made, the DCT is based on the temporal Prediction error applied.

B-pictures: With the B-pictures the temporal becomes bidirectional Prediction error calculated and then transformed. The bidirectional prediction basically works adaptively, d. H. it will be one Forward prediction, a backward prediction or an interpolation allowed.

An image sequence is used in MPEG-2 coding in so-called GOPs (Group Of Pictures). n pictures from an I picture to next form a GOP. The distance between the P-pictures is denoted by m, where there are m-1 B images between the P-pictures. The MPEG syntax leaves it up to you however, the user how m and n are chosen. m = 1 means that no B-pictures are used and n = 1 means that only I pictures are encoded.

From [2] is a method for motion estimation in the frame a method for block-based image coding known. It is assumed that a digitized image Has pixels that are in particular 8 × 8 image blocks Pixels or 16 × 16 pixels are combined. If necessary, one picture block can also have several picture blocks include. An example of this is a macro block with 6 Picture blocks, of which 4 picture blocks for  Brightness information and 2 image blocks for color information are provided.

In the case of a sequence of pictures, the following is done for a picture to be coded, taking into account the picture blocks of this picture:

• - A value for an error measure is determined for the image block for which a motion estimation is to be carried out in a temporally preceding image, starting from an image block which was in the same relative position in the previous image (= previous image block) . For this purpose, a sum is preferably determined via the amounts of the differences from the coding information associated with the pixels of the image block and the previous image block.
  Coding information here means brightness information (luminance value) and / or color information (chrominance value), each of which is assigned to a pixel.
• - In a search space of predeterminable size and shape Starting position in the previous picture is used for an area of the same size previous image blocks, shifted by one or half a pixel, a value of the error measure certainly.
• - In a search space of the size n × n pixels, there are n ² (error) values. The shifted previous image block in the temporally preceding image is determined for which the error measure gives a minimum error value. For this picture block, it is assumed that this previous picture block best matches the picture block of the picture to be coded, for which the motion estimation is to be performed.
• - The result of the motion estimation is a Motion vector with which the shift between the picture block in the picture to be coded and the  selected image block in the secondary previous picture is described.
• - A compression of the image data is achieved that encodes the motion vector and the error signal become.
• - In particular, the motion estimation for everyone Image block of an image performed.

The application of the motion estimation within the block-based or object-based image coding is in [3] described.

It is general in image data - in compressed image data in particular - almost impossible to search for content. Such a search should focus on any objects in the Image sequences that are not available in descriptive form, but are only part of the image data stream.

The object of the invention is an image data stream regarding the information contained in the image data to make it searchable.

This task is carried out according to the characteristics of the independent Claims resolved. Developments of the invention result itself from the dependent claims.

A method for saving is used to solve the problem at least one image specified by a computer at the one relational information with the at least one Image is saved associated.

This relational information can be used together with the at least one image can be saved. Alternatively can be a reference (pointer, pointer) to the relational Information can be saved together with the picture.  

A further training is that the relational Information is determined before saving.

It is also a further training that the relational Information a feature information and a Reference information between objects and / or images includes. The feature information represents z. B. Information about represents a movement feature by which reference information is the link to the object or image for which the feature information is relevant.

In particular, the relational information marks information regarding a given relationship between two objects, on the one hand the information for the type of relationship (feature information) and the at the Objects involved (reference information) in the relationship relational information can be summarized.

It should be noted that the association of relational information can be realized with the image in this way may be a reference to relational information is saved. It is not necessary to do the same Use memory for information and image data. A division over any storage space is possible, a link information (pointer) is preferred stored, based on which the actual information can be found.

A further training is that the at least one Image is a sequence of multiple images.

Accordingly, when saving images, an additional one Information, referred to here as relational information, determined and saved with the pictures. The following explained type of relational information enables a later search for specific image data. The search takes place preferably in the data of the relational information (s) instead, the image data, preferably in compressed form  do not need to be specially restored. Based on this search at a high level of abstraction "red car drives from left to right through the picture ", can be targeted certain pictures, here the red car, can be found. A Such search option in (especially compressed) Image data did not yet exist.

Image compression can be done especially after a Image compression standard, e.g. B. an MPEG or an H.26x Standard.

A further training is that the relational Information at least one of the following options includes:

a) Movement information

As stated in [2], image data can be generated automatically Movement information (especially between objects) be determined. As of MPEG-4, objects are in one picture identifiable, the picture itself is hierarchical (comparable to a tree structure). The hierarchical relationships of objects to each other can movement information between the respective objects be supplemented. This movement information marks the relative movement of the connected objects. Here the total movement (relative and absolute) of all results in the image of existing and relevant objects by the complete hierarchical structure. The hierarchical Structure of the picture (or scene) can be made according to different requirements: An example is a "contain-in" relation, that is the hierarchical Structure specifies which objects in other objects (at least partially) are included. Others too Examples of a hierarchical division of the scene are possible.  

b) Distance information

Instead of or in addition to the movement information determines the distance information between objects and can be saved. The distance can be, for example based on a boundary or a focus of a Object can be determined. Based on the distances between the several objects in the scene are complete described.

c) Overlap information

With the overlap information is called relational Information the type or degree of overlap between Objects. The sum of the overlaps is the Arrangement of objects within the scene.

d) Information regarding a relationship between objects and / or pictures

In general, any relationship between objects and / or Images can be used as relational information. The hierarchical arrangement of the objects described above A scene can be chosen according to the chosen relationship respectively.

Especially when it comes to motion information The following parameters are recorded: Translation (along the Coordinate axis (s)), rotation and zoom (enlargement / ver reduction) of the object.

Furthermore, a Serve transformation information over time. In one In this case, objects / images are preferred over a given one Time period transformed, the transformation values gives the mean values of the movement over time. Such an average is obtained, for example, using discrete Cosine transformation (DCT).  

It should be noted that the relational Information especially between two pictures or between two objects in a picture below Taking into account the change in relational information over time (e.g. movement information) can be determined can.

As already mentioned above, an image or a Scene include a variety of objects that are in each other Connected and their position over time changed differently. The relational information can between two objects according to their hierarchical Arrangement can be determined. Alternatively, the relational Information also based on absolute information (e.g. absolute Coordinates within the image) can be determined. From the absolute information is the information of the objects to each other and vice versa.

In one embodiment, the relational information becomes a feature set after an image compression process added. The image compression process is included especially standardized. Examples represent an MPEG Standard or an H.26x standard.

The method described can preferably be used as part of encoding using an image compression process.

A further development is that image data based on the method described were stored selectively can be accessed by using appropriate Search mechanisms that relational information is implemented. For example, the motion information between objects that are in the feature set is stored, searched specifically and being found. The aforementioned search for the red one Car moving from left to right through a picture is possible.  

It should be noted that the search itself based on different functionality with the procedure can use the relational information described. So is an "intelligent" evaluation of the different Information within an application-specific defined Search possible. Relational information alone makes it possible the search in image data that would otherwise not be searchable Features.

To solve the problem, an arrangement for Saving of at least one image specified, in which one Processor unit is provided, which is set up in such a way that relational information with the at least one Image associated can be saved.

This arrangement is particularly suitable for implementation of the method according to the invention or one of its above explained further training.

Exemplary embodiments of the invention are described below shown and explained in the drawing.

Show it

Fig. 1 is a scene that is hierarchically divided into three objects;

Fig. 2 is an image sequence showing a scene over time;

Fig. 3 is a possibility of object-related for storing image data;

Figure 4 is a sketch illustrating a transmitter and receiver for image compression;

Figure 5 is a diagram showing an image coder and an image decoder in greater detail.

Fig. 6 is a processor unit;

Fig. 7 shows an alternative embodiment of object-related for storing image data.

In Fig. 1, a hierarchical structure of a square 101, a rectangle 102 and the triangle 103 in shape is shown a tree diagram. The connections 104 and 105 between square 101 and rectangle 102 or between square 101 and triangle 103 correspond to an “included-in” relationship, ie square 101 contains both rectangle 102 and triangle 103 .

This relationship is illustrated with the aid of FIG. 2.

FIG. 2 comprises a scene which is shown in different versions 201 , 202 , 203 and 204 over time. The objects of the hierarchical structure of FIG. 1 are present in every temporal version of the scene.

Thus, the square 101 moves from its starting position 205 down to the left 206 , further down 207 and then to the right 208 . Within the square 101 , the rectangle 102 remains in the unchanged position at the top left during the changes over time (indicated by the arrows 217 , 218 and 219 ) (see positions 209 , 210 , 211 and 212 ). The triangle 103 is also contained in the square 101 and moves gradually from an initial position 213 in the different time steps 217 to 219 (see positions 214 , 215 and 216 ).

The relations 104 and 105 from FIG. 1 can thus be expanded per time step by the movement information of the two connected objects square 101 and rectangle 102 (for the connection 104 ) and square 101 and triangle 103 (for the connection 105 ). The relative change in position per time step is preferably specified by means of the parameters translation (along the coordinate axes), rotation and zoom.

This can also include methods for motion estimation, such as mentioned in the introduction.

Fig. 3 shows a possibility for storing image data, in particular a division of an image into objects, for. B. according to the MPEG-4 standard. A sequence 301 of image data for an object 1 and a sequence 302 of image data for an object 2 are shown. The relational information, in particular movement information corresponding to the above explanations, is per object within a set of features that contains both intrinsic data 303 and 305 (e.g. B. shape and color of the object) as well as relational information 304 or 306 .

The relation is preferably also supplemented by a reference 315 or 316 (pointer). This reference represents the linking of the hierarchically structured objects. In the example of FIG. 1, object 1 corresponds to square 101 and object 2 to rectangle 102 . Arrow 316 indicates the relation "contains" and arrow 315 indicates the relation "contained in". The change in position between object 1 and object 2 for sequences 301 and 302 is also stored in the fields for relational information 304 and 306, respectively.

The object-related data 307 to 310 (for object 1 ) or 311 to 314 (for object 2) determine the respective sequences 301 and 302 . Relational information is determined and stored for these sequences, in particular each sequence is interpreted as a "global" movement. ie a set of features ( 303 and 304 or 305 and 306 ) is determined and stored for the entirety of a sequence.

FIG. 4 shows an arrangement which comprises two computers and a camera, with image coding, transmission of the image data and image decoding being illustrated.

A camera 1101 is connected to a first computer 1102 via a line 1119 . The camera 1101 transmits captured images 1104 to the first computer 1102 . The first computer 1102 has a first processor unit 1103 , which is connected via a bus 1118 to an image memory 1105 . The image coding methods are carried out with the processor unit 1103 of the first computer 1102 . Image data 1106 encoded in this way is transmitted from the first computer 1102 to a second computer 1108 via a communication link 1107 , preferably a line or a radio link. The second computer 1108 contains a second processor unit 1109 which is connected to the image memory 1111 via a bus 1110 . Methods for image decoding are carried out on the second processor unit 1109 .

Both the first computer 1102 and the second computer 1108 each have a screen 1112 or 1113 on which the image data 1104 are visualized. Input units are provided for operating both the first computer 1102 and the second computer 1108 , preferably a keyboard 1114 or 1115 , and a computer mouse 1116 or 1117 .

The image data 1104 , which are transmitted from the camera 1101 via the line 1119 to the first computer 1102 , are preferably data in the time domain, while the data 1106 , which are transmitted from the first computer 1102 to the second computer 1108 via the communication link 1107 , Image data are in the spectral range.

The decoded image data is displayed on a screen 1120 .

FIG. 5 shows a sketch of an arrangement for carrying out a block-based image coding method.

A video data stream to be encoded with chronologically successive digitized images is fed to an image coding unit 1201 . The digitized images are divided into macro blocks 1202 , each macro block having 16 × 16 pixels. The macro block 1202 comprises 4 picture blocks 1203 , 1204 , 1205 and 1206 , each picture block containing 8 × 8 picture elements to which luminance values (brightness values) are assigned. Furthermore, each macro block 1202 comprises two chrominance blocks 1207 and 1208 with chrominance values (color information, color saturation) assigned to the pixels.

The block of an image contains a luminance value (= Brightness), a first chrominance value (= hue) and a second chrominance value (= color saturation). In doing so Luminance value, first chrominance value and second Chrominance value referred to as color values.

The image blocks are fed to a transformation coding unit 1209 . In the case of differential image coding, values to be coded from image blocks of temporally preceding images are subtracted from the image blocks currently to be coded; only the difference formation information 1210 is supplied to the transformation coding unit (Discrete Cosine Transformation, DCT) 1209 . For this purpose, the current macro block 1202 is communicated to a motion estimation unit 1229 via a connection 1234 . Spectral coefficients 1211 are formed in the transformation coding unit 1209 for the picture blocks or difference picture blocks to be coded and are fed to a quantization unit 1212 . This quantization unit 1212 corresponds to the quantization device according to the invention.

Quantized spectral coefficients 1213 are supplied to both a scan unit 1214 and an inverse quantization unit 1215 in a reverse path. After a scanning process, e.g. For example, a "zigzag" scanning method, entropy coding is carried out on the scanned spectral coefficients 1232 in an entropy coding unit 1216 provided for this purpose. The entropy-coded spectral coefficients are transmitted as coded image data 1217 via a channel, preferably a line or a radio link, to a decoder.

The inverse quantization unit 1215 inversely quantizes the quantized spectral coefficients 1213 . Spectral coefficients 1218 obtained in this way are fed to an inverse transformation coding unit 1219 (inverse discrete cosine transformation, IDCT).

Reconstructed coding values (also differential coding values) 1220 are supplied to an adder 1221 in the differential image mode . The adder 1221 also receives coding values of an image block which result from a temporally preceding image after motion compensation has already been carried out. Reconstructed image blocks 1222 are formed with the adder 1221 and stored in an image memory 1223 .

Chrominance values 1224 of the reconstructed image blocks 1222 are fed from the image memory 1223 to a motion compensation unit 1225 . For brightness values 1226 , an interpolation takes place in an interpolation unit 1227 provided for this purpose. The number of brightness values contained in the respective image block is preferably doubled on the basis of the interpolation. All brightness values 1228 are supplied to both the motion compensation unit 1225 and the motion estimation unit 1229 . The motion estimation unit 1229 also receives the picture blocks of the macro block to be coded in each case (16 × 16 picture elements) via the connection 1234 . In the motion estimation unit 1229 , the motion estimation takes into account the interpolated brightness values ("motion estimation on a half-pixel basis"). When estimating the movement, absolute differences between the individual brightness values are preferably determined in the macro block 1202 that is currently to be coded and in the reconstructed macro block from the previous image. The result of the motion estimation is a motion vector 1230 , by means of which a local shift of the selected macroblock from the temporally preceding image to the macroblock 1202 to be coded is expressed.

Both brightness information and chrominance information relating to the macroblock determined by the motion estimation unit 1229 are shifted by the motion vector 1230 and subtracted from the coding values of the macroblock 1202 (see data path 1231 ).

In FIG. 6, a processor unit PRZE. The processor unit PRZE comprises a processor CPU, a memory SPE and an input / output interface IOS, which is used in different ways via an interface IFC: an output is visible on a monitor MON and / or on a printer via a graphic interface PRT issued. An entry is made using a mouse MAS or a keyboard TAST. The processor unit PRZE also has a data bus BUS, which ensures the connection of a memory MEM, the processor CPU and the input / output interface IOS. Furthermore, additional components can be connected to the data bus BUS, e.g. B. additional memory, data storage (hard disk) or scanner.

FIG. 7 shows an alternative embodiment to FIG. 3 for storing object-related image data. A sequence 701 of image data for object 1 and a sequence 702 image data for object 2 are shown. Intrinsic information (shape, color of the object) belonging to the respective object 703 or 704 is stored with object 701 or 702, respectively . The relational information 713 is preferably stored separately from the respective objects 701 and 702 . Relational information 713 includes feature information 714 relating to the linking of objects 701 and 702 , e.g. B. The movement of the object 1 relative to the object 2. The link itself is established by means of the reference information 715 , 716 , which preferably has references to the objects 701 and 702 associated with the feature information 714 .

The object-related data 705 to 708 or 704 to 712 each determine a sequence belonging to the respective object. The sequence can include any number of images (for the respective object).

Bibliography:
[1] J. De Lameillieure, R. Schäfer: "MPEG-2 image coding for digital television", television and cinema technology, 48th year, No. 3/1994, pages 99-107.
[2] M. Bierling: "Displacement Estimation by Hierarchical Blockmatching", SPIE, Vol: 1001, Visual Communications and Image Processing '88, pp. 942-951, 1988.
[3] ITU-T, International Telecommunication Union, Telecommunications Sector of ITU, Draft ITU-T Recommendation H.263, Videocoding for low bitrate communication, 2.5.96.

Claims (15)

1. Method for storing at least one image a calculator, where a relational information with the at least an image is saved associated. 2. The method according to claim 1, where the relational information on the at least an image is determined. 3. The method according to any one of the preceding claims, where the relational information is a Feature information and related information between Objects and / or images includes. 4. The method according to any one of the preceding claims, in which the at least one image is a sequence of multiple pictures. 5. The method according to any one of the preceding claims, wherein the relational information comprises at least one of the following options:

• a) movement information;
• b) distance information;
• c) overlap information;
• d) Information regarding a relationship between objects and / or images.

6. The method according to claim 5, wherein the movement information comprises the following parameters:

• a) translation,
• b) rotation,
• c) Zoom.

7. The method according to any one of the preceding claims, where the relational information is based on a Transformation over time is determined. 8. The method according to any one of the preceding claims, where the relational information about two Images is determined. 9. The method according to any one of the preceding claims, in which the at least one image has at least two objects contains for which the relational information is intended becomes. 10. The method according to claim 9, where the relational information about two Objects is determined. 11. The method according to any one of the preceding claims, where the relational information about a feature set is added according to an image compression standard. 12. The method according to claim 11, where the image compression standard is an MPEG standard or is an H.26x standard. 13. The method according to any one of the preceding claims for use in encoding after a Image compression process. 14. The method according to any one of the preceding claims, in the case of the with the at least one picture stored relational information a search in the stored data is made possible. 15. Arrangement for storing at least one image a calculator,  in which a processor unit is provided which is set up that relational information with the at least one Image associated can be saved.