CN113781493A - Image processing method, apparatus, electronic device, medium and computer program product - Google Patents
Image processing method, apparatus, electronic device, medium and computer program product Download PDFInfo
- Publication number
- CN113781493A CN113781493A CN202110002321.5A CN202110002321A CN113781493A CN 113781493 A CN113781493 A CN 113781493A CN 202110002321 A CN202110002321 A CN 202110002321A CN 113781493 A CN113781493 A CN 113781493A
- Authority
- CN
- China
- Prior art keywords
- image
- map
- segmentation
- processing
- predicted
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
本公开实施例提供了一种图像处理方法、装置、电子设备、介质及计算机程序产品。该方法包括：获取目标图像，其中，目标图像包括目标对象和非目标对象；对目标图像进行图像分割处理和深度估计处理，分别得到目标图像的预测分割图和预测深度图；根据目标对象的预测分割图确定目标图像的预测深度图中目标对象的位置；根据目标图像的预测深度图中目标对象的位置对预测深度图进行处理，得到目标对象的预测深度图。
Embodiments of the present disclosure provide an image processing method, apparatus, electronic device, medium, and computer program product. The method includes: acquiring a target image, wherein the target image includes a target object and a non-target object; performing image segmentation processing and depth estimation processing on the target image to obtain a predicted segmentation map and a predicted depth map of the target image respectively; The segmentation map determines the position of the target object in the predicted depth map of the target image; the predicted depth map is processed according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object.
Description
Technical Field
The disclosed embodiments relate to the field of computer technologies, and in particular, to an image processing method, an image processing apparatus, an electronic device, a medium, and a computer program product.
Background
In the field of computer vision, depth estimation is part of three-dimensional reconstruction, which requires estimating depth information from two-dimensional images. For some specific tasks, such as a monocular robot avoiding or finding a target object, the target object may be a human, segmenting the target object from the two-dimensional image, and estimating the depth of the target object has an extremely important role.
In implementing the disclosed concept, the inventors found that there are at least the following problems in the related art: depth estimation for a target object in a target image is difficult to achieve with related techniques.
Disclosure of Invention
In view of this, the disclosed embodiments provide an image processing method, an image processing apparatus, an electronic device, a medium, and a computer program product.
An aspect of an embodiment of the present disclosure provides an image processing method, including: acquiring a target image, wherein the target image comprises a target object and a non-target object; carrying out image segmentation processing and depth estimation processing on the target image to respectively obtain a predicted segmentation map and a predicted depth map of the target image; determining the position of the target object in the predicted depth map of the target image according to the predicted segmentation map of the target object; and processing the predicted depth map according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object.
Another aspect of an embodiment of the present disclosure provides an image processing apparatus including: the device comprises an acquisition module, a display module and a processing module, wherein the acquisition module is used for acquiring a target image, and the target image comprises a target object and a non-target object; the first processing module is used for carrying out image segmentation processing and depth estimation processing on the target image to respectively obtain a predicted segmentation map and a predicted depth map of the target image; a determining module, configured to determine a position of the target object in a predicted depth map of the target image according to the predicted segmentation map of the target object; and the second processing module is used for processing the predicted depth map according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object.
Another aspect of an embodiment of the present disclosure provides an electronic device including: one or more processors; memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method as described above.
Another aspect of embodiments of the present disclosure provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method as described above.
Another aspect of an embodiment of the present disclosure provides a computer program product comprising a computer program for implementing the method as described above when executed by a processor.
According to the embodiment of the disclosure, a target image is obtained, the target image comprises a target object and a non-target object, image segmentation processing and depth estimation processing are carried out on the target image, a predicted segmentation map and a predicted depth map of the target image are obtained respectively, the position of the target object in the predicted depth map of the target image is determined according to the predicted segmentation map of the target object, and the predicted depth map is processed according to the position of the target object in the predicted depth map of the target image, so that the predicted depth map of the target object is obtained. The image segmentation and the depth estimation are combined, wherein the position of the target object in the predicted depth map can be obtained according to the predicted segmentation map, and the predicted depth map of the target object can be obtained by processing the predicted depth map according to the position of the target object in the predicted depth map, so that the technical problem that the depth estimation of the target object in the target image is difficult to realize in the related art is at least partially overcome, the depth of the target object in the target image is accurately determined, and the generalization of the method is strong.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:
fig. 1 schematically illustrates an exemplary system architecture to which the image processing method and apparatus of the embodiments of the present disclosure may be applied;
FIG. 2 schematically shows a flow chart of a method of image processing according to an embodiment of the present disclosure;
FIG. 3 schematically illustrates an image processing model structure diagram according to an embodiment of the disclosure;
FIG. 4 schematically shows a flow chart of another image processing method according to an embodiment of the present disclosure;
FIG. 5 schematically shows a schematic view of a target image according to an embodiment of the disclosure;
FIG. 6 schematically illustrates a predicted depth map of a target image according to an embodiment of the disclosure;
FIG. 7 schematically illustrates a predictive segmentation map of a target image according to an embodiment of the disclosure;
FIG. 8 schematically illustrates a predicted depth map of a target object according to an embodiment of the disclosure;
FIG. 9 schematically illustrates another predicted depth map of a target object, in accordance with an embodiment of the present disclosure;
FIG. 10 schematically illustrates a schematic view of yet another target object according to an embodiment of the present disclosure;
FIG. 11 schematically illustrates a flow chart of yet another image processing method according to an embodiment of the present disclosure;
FIG. 12 schematically shows a flow chart of yet another image processing method according to an embodiment of the present disclosure;
fig. 13 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure; and
fig. 14 schematically shows a block diagram of an electronic device suitable for an image processing method according to an embodiment of the present disclosure.
Detailed Description
Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is illustrative only and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. Moreover, in the following description, descriptions of well-known structures and techniques are omitted so as to not unnecessarily obscure the concepts of the present disclosure.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.
All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It is noted that the terms used herein should be interpreted as having a meaning that is consistent with the context of this specification and should not be interpreted in an idealized or overly formal sense.
Where a convention analogous to "at least one of A, B and C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B and C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a convention analogous to "A, B or at least one of C, etc." is used, in general such a construction is intended in the sense one having skill in the art would understand the convention (e.g., "a system having at least one of A, B or C" would include but not be limited to systems that have a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).
The embodiment of the disclosure provides an image processing method, an image processing device and an electronic device applying the method. The method includes acquiring a target image, wherein the target image includes a target object and a non-target object. And carrying out image segmentation processing and depth estimation processing on the target image to respectively obtain a predicted segmentation map and a predicted depth map of the target image. And determining the position of the target object in the predicted depth map of the target image according to the predicted segmentation map of the target object. And processing the predicted depth map according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object.
Fig. 1 schematically illustrates an exemplary system architecture 100 to which an image processing method or apparatus may be applied, according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which the embodiments of the present disclosure may be applied to help those skilled in the art understand the technical content of the present disclosure, and does not mean that the embodiments of the present disclosure may not be applied to other devices, systems, environments or scenarios.
As shown in fig. 1, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired and/or wireless communication links, and so forth.
The user may use the terminal devices 101, 102, 103 to interact with the server 105 via the network 104 to receive or send messages or the like. Various messaging client applications, such as image processing applications, model building applications, search applications, instant messaging tools, mailbox clients, and/or social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.
The terminal devices 101, 102, 103 may be various electronic devices having a display screen and supporting image processing, web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, desktop computers, and the like.
The server 105 may be a server that provides various services, such as a background management server (for example only) that provides support for websites, processed pictures, etc. that users browse with the terminal devices 101, 102, 103. The background management server can process, analyze, store and the like the received data such as the images and the camera information, and feed back the processing result (such as the webpage, the information, the data and the like acquired or generated according to the user request) to the terminal equipment.
It should be noted that the image processing method provided by the embodiment of the present disclosure may be generally executed by the server 105. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may be generally disposed in the server 105. The image processing method provided by the embodiment of the present disclosure may also be executed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Alternatively, the image processing method provided by the embodiment of the present disclosure may also be executed by the terminal device 101, 102, or 103, or may also be executed by another terminal device different from the terminal device 101, 102, or 103. Accordingly, the image processing apparatus provided by the embodiment of the present disclosure may also be disposed in the terminal device 101, 102, or 103, or in another terminal device different from the terminal device 101, 102, or 103.
For example, the target image may be originally stored in any one of the terminal apparatuses 101, 102, or 103 (e.g., the terminal apparatus 101, but not limited thereto), or stored on an external storage apparatus and may be imported into the terminal apparatus 101. Then, the terminal device 101 may locally execute the image processing method provided by the embodiment of the present disclosure, or transmit the target image to another terminal device, server, or server cluster, and execute the image processing method provided by the embodiment of the present disclosure by another terminal device, server, or server cluster that receives the target image.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
Fig. 2 schematically shows a flow chart of an image processing method according to an embodiment of the present disclosure.
As shown in fig. 2, the method includes operations S210 to S240.
In operation S210, a target image is acquired, wherein the target image includes a target object and a non-target object.
According to the embodiment of the present disclosure, the target image may be a monocular image, and the target image may include a target object and a non-target object, wherein the target object may be a person in the target image, and the non-target object may be a background object in the target image, such as a table, a tree, a car, and the like, but not limited thereto, any target may be designated as the target object according to actual needs, and all objects different from the designated target object are the non-target objects. The number of target objects may include one or more.
In operation S220, an image segmentation process and a depth estimation process are performed on the target image to obtain a predicted segmentation map and a predicted depth map of the target image, respectively.
According to the embodiment of the disclosure, the depth estimation processing is performed on the target image, and the depth estimation can be performed on each pixel point on the target image according to the depth relation reflected by the pixel value relation.
According to an embodiment of the present disclosure, the image segmentation may be a method of semantic segmentation, but is not limited thereto, and may also be a method of example segmentation. Semantic segmentation can be performed on each pixel point in an image by category division to obtain a semantic segmentation mask which has a corresponding relation with the image size, namely a predicted semantic segmentation map of the image, but the semantic segmentation does not distinguish different objects in the same category, namely examples. The example segmentation can not only realize the classification of the pixel points, but also distinguish different objects in the same class, namely, distinguish the examples, and an example segmentation mask which has a corresponding relation with the image size, namely, a prediction example segmentation graph of the image can be obtained by carrying out the example segmentation on the image. The prediction semantic segmentation map of the image and the prediction instance segmentation map of the image may be collectively referred to as a prediction segmentation map of the image. In embodiments of the present disclosure, different classes may be characterized by different colors, and correspondingly, different colors in the predictive segmentation map of the image characterize different classes.
According to the embodiment of the disclosure, semantic segmentation or example segmentation can be adopted to perform image segmentation processing on the target image, so as to obtain the predicted segmentation map of the target image. The size of the predicted segmentation map of the target image may be determined according to actual conditions, and is not particularly limited herein. For example, the predicted segmentation map size of the target image is the same as the target image size. Alternatively, the predicted segmentation map size of the target image is one-half of the target image size.
According to the embodiment of the disclosure, the image segmentation processing by adopting the semantic segmentation method can be realized in real time at a high speed, so that the processing requirement of a real-time task is met.
In operation S230, a position of the target object in the predicted depth map of the target image is determined according to the predicted segmentation map of the target object.
According to the embodiments of the present disclosure, the position of the target object in the predicted partition map of the target image may be determined according to the predicted partition map of the target object in the predicted partition map of the target image, and since the position of the target object in the predicted partition map of the target image corresponds to the position of the target object in the predicted depth map of the target image, the position of the target object in the predicted depth map of the target image may be obtained based on the position of the target object in the predicted partition map of the target image.
In operation S240, the predicted depth map of the target object is obtained by processing the predicted depth map according to the position of the target object in the predicted depth map of the target image.
According to an embodiment of the present disclosure, in a related art implementing the present disclosure, a depth estimation may be performed on a monocular image in combination with a gradient and a texture feature. The gradient information and the texture information of the target image can be used as a depth clue to assist the depth convolution network to learn the depth information of the target image, and the predicted depth map of the target image is obtained. The target image may also be input to a depth estimation network based on camera pose estimation to obtain a predicted depth map of the target image.
However, the above methods can only perform depth estimation on the entire target image, and it is difficult to extract the depth of the target object in the target image.
In a related art implementing the present disclosure, a sample-based learning method may also be employed to achieve depth estimation. The learning method based on the samples is to construct a data set, convert the depth estimation problem of the target object into a retrieval problem, and retrieve in the data set by adopting a feature matching method to obtain the depth estimation result of the target object in the target image. The sample-based learning method is capable of estimating the depth of a target object in a target image having a matching relationship with an image in a data set. But if an image matching the target image cannot be retrieved in the dataset, depth estimation for the target object in the target image cannot be achieved. In addition, the depth of the target object in the target image can be estimated by adopting the method, the position of the target object in the target image cannot be obtained, and the method has poor generalization and low estimation precision.
According to the embodiment of the disclosure, since the image segmentation and the depth estimation are combined, wherein the position of the target object in the predicted depth map of the target image can be obtained according to the predicted segmentation map of the target image, and the predicted depth map of the target object can be obtained by processing the predicted depth map according to the position of the target object in the predicted depth map of the target image, the depth of the target object in the target image can be accurately determined. In addition, because the depth estimation which is not realized by the way of searching matching is less influenced by the sample, the generalization of the scheme of the embodiment of the disclosure is stronger.
It should be noted that, if the number of the target objects includes at least two, the method provided in the above operations S210 to S240 may be used for each target object to obtain the predicted depth map of the target object. In the presentation, the predicted depth maps of the respective target objects may be presented on one predicted depth map, or the predicted depth maps of each target object may be presented individually.
According to the technical scheme of the embodiment of the disclosure, the target image is obtained and comprises a target object and a non-target object, the target image is subjected to image segmentation processing and depth estimation processing to respectively obtain a predicted segmentation map and a predicted depth map of the target image, the position of the target object in the predicted depth map of the target image is determined according to the predicted segmentation map of the target object, and the predicted depth map is processed according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object. The image segmentation and the depth estimation are combined, wherein the position of the target object in the predicted depth map can be obtained according to the predicted segmentation map, and the predicted depth map of the target object can be obtained by processing the predicted depth map according to the position of the target object in the predicted depth map, so that the technical problem that the depth estimation of the target object in the target image is difficult to realize in the related art is at least partially overcome, the depth of the target object in the target image is accurately determined, and the generalization of the method is strong.
The method shown in fig. 2 is further described with reference to fig. 3-10 in conjunction with specific embodiments.
FIG. 3 schematically shows a diagram of an image processing model architecture according to an embodiment of the disclosure.
As shown in fig. 3, the image processing model includes a feature extraction network, an image segmentation network, and a depth estimation network.
According to the embodiment of the disclosure, an image processing model is constructed by training an encoding and decoding network by using a training sample, namely, a target image to be predicted is input into the image processing model, a prediction segmentation map and a prediction depth map of the target image are respectively output, and then a prediction depth map of a target object of the target image is obtained. According to the embodiment of the disclosure, the image processing model constructed by the disclosure makes up for the technical deficiency that the prediction segmentation map and the prediction depth map containing the target image cannot be output simultaneously in the related technology.
Fig. 4 schematically shows a flow chart of another image processing method according to an embodiment of the present disclosure.
As shown in fig. 4, processing the target image using the image processing model to obtain the predicted segmentation map and the predicted depth map of the target image respectively may include the following operations S410 to S460.
In operation S410, the target image is processed using the feature extraction network to obtain a first intermediate feature map.
In operation S420, the first intermediate feature map is processed using an image segmentation network to obtain a second intermediate feature map.
In operation S430, the first intermediate feature map is processed by using the depth estimation network to obtain a third intermediate feature map.
In operation S440, a fourth intermediate feature map is generated according to the second intermediate feature map and the third intermediate feature map.
In operation S450, the fourth intermediate feature map is processed by using the depth estimation network to obtain a predicted depth map of the target image.
In operation S460, the second intermediate feature map is processed using the image segmentation network to obtain a predicted segmentation map of the target image.
As shown in fig. 3 and 4, in order to achieve the depth estimation of the target object in real time, a MobileNet + ASPP (empty Spatial convolution Pooling Pyramid) module is used as a feature extraction network, i.e., a coding network. Depth separable convolutions are used as decoding networks, i.e. an image segmentation network and a depth estimation network.
According to an embodiment of the present disclosure, the height and width of the target image may be written as H and W, respectively. Inputting the target image into a MobileNet module and outputting a characteristic diagram f2Of a size of
In the process, the output of the intermediate layer is a characteristic diagram f1Of a size of
The characteristic diagram f can be combined2Inputting the output and the characteristic diagram f into an ASPP module2Fusing and outputting the characteristic diagram f3. Will the characteristic diagram f3Up-sampling to
Size, then the feature map f1And f3Merging, finally using characteristic diagram f4As output of the feature extraction network, feature map f4Namely the first intermediate characteristic diagram.
According to the embodiment of the disclosure, the image segmentation network and the depth estimation network are both provided with the same feature map f4(i.e. the first intermediate characteristic diagram) as input, and respectively outputting a second intermediate characteristic diagram (i.e. the characteristic diagram f)5) And a third intermediate feature map (i.e., feature map f)6). It should be noted that, since the first intermediate feature map processed by the feature extraction network is used as the input of the depth estimation network and the image segmentation network, the feature extraction network comprehensively considers the detail information and the abstract information of the target image.
According to the embodiment of the disclosure, since the depth values of the same target object are relatively close in the predicted depth map, and the gradient of the depth values may be larger at the boundary of the target object, in order to obtain a predicted depth map with higher precision, the second intermediate feature map (i.e. the feature map f) output by the image segmentation network may be output5) Inputting the data into a depth estimation network and comparing the data with a third intermediate feature map (i.e. feature map f)6) Combining to obtain a fourth intermediate characteristic diagram (i.e. characteristic diagram f)7). The fourth intermediate feature map is obtained by convolution and upsampling to have a size of
The predicted depth map of (1). The second intermediate feature map is obtained by convolution and upsampling to have a size of
Is predicted for the segmentation map.
According to the embodiment of the disclosure, the second intermediate feature map in the image segmentation network is input into the depth estimation network to correct the prediction result of the depth estimation network, so that the prediction result of the depth estimation is more accurate.
According to the embodiment of the disclosure, processing the predicted depth map according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object may include the following operations.
And setting the pixel values of other positions except the position of the target image in the predicted depth map of the target image as preset pixel values to obtain the predicted depth map of the target object.
According to the embodiment of the disclosure, the predicted depth map of the target object can be obtained by setting the pixel values of other positions in the predicted depth map of the target image except the position of the target image as the preset pixel values. The preset pixel value may be set according to actual conditions, and is not limited herein, for example, the preset pixel value may be 0.
Illustratively, fig. 5 schematically shows a schematic view of a target image according to an embodiment of the present disclosure. Fig. 6 schematically shows a predicted depth map of a target image according to an embodiment of the present disclosure. FIG. 7 schematically illustrates a predictive segmentation map of a target image according to an embodiment of the disclosure. Fig. 8 schematically illustrates a predicted depth map of a target object according to an embodiment of the present disclosure. The target object in fig. 8 is a human. Fig. 9 schematically illustrates a predicted depth map of another target object according to an embodiment of the present disclosure. The target object in fig. 9 is a refrigerator. Fig. 10 schematically shows a schematic view of a further target object according to an embodiment of the present disclosure. The number of target objects in fig. 10 includes plural.
As shown in fig. 5 to 10, the predicted depth map of the target object is obtained by adding the predicted segmentation map of the target image to the predicted depth map of the target image. In fig. 7 to 10, black indicates a non-target region.
According to the embodiment of the disclosure, an image processing model is used for processing a target image to obtain a prediction segmentation map and a prediction depth map of the target image respectively, wherein the image processing model is obtained by training with a training sample, and the training sample comprises a sample image and a depth label and a segmentation label of the sample image.
According to an embodiment of the present disclosure, the image processing model is trained by using training samples, and may include the following operations.
Training samples are obtained. And training the full convolution neural network model by using the training sample to obtain an image processing model.
According to an embodiment of the present disclosure, a full convolution neural network model includes an initial feature extraction network, an initial image segmentation network, and an initial depth estimation network.
According to an embodiment of the present disclosure, training a full convolution neural network model with a training sample to obtain an image processing model may include the following operations.
And utilizing the initial feature extraction network to process the sample image to obtain a fifth intermediate feature map. And processing the fifth intermediate feature map by using the initial image segmentation network to obtain a sixth intermediate feature map. And processing the fifth intermediate feature map by using the initial depth estimation network to obtain a seventh intermediate feature map. And generating an eighth intermediate feature map according to the sixth intermediate feature map and the seventh intermediate feature map. And processing the eighth intermediate feature map by using the initial depth estimation network to obtain a predicted depth map of the sample image. And processing the sixth intermediate characteristic graph by using the initial image segmentation network to obtain a prediction segmentation graph of the sample image. And inputting the depth label, the predicted depth map, the segmentation label and the predicted segmentation map of the sample image into a loss function of the full convolution neural network model, and outputting a loss result. And adjusting the network parameters of the full convolution neural network model according to the loss result until the loss function is converged. And taking the trained full convolution neural network model as an image processing model.
According to an embodiment of the disclosure, as can be understood in conjunction with fig. 3, in the training process of the image processing model, the fifth intermediate feature map may be understood as the feature map f in fig. 34The sixth intermediate characteristic diagram can be understood as the characteristic diagram f in fig. 35The seventh intermediate characteristic diagram can be understood as the characteristic diagram f in fig. 36The eighth intermediate feature map can be understood as the feature map f in fig. 37. After the training is completed, the initial feature extraction network, the initial image segmentation network, and the initial depth estimation network are respectively referred to as a feature extraction network and an image segmentation networkNetworks and depth estimation networks.
Fig. 11 schematically shows a flow chart of yet another image processing method according to an embodiment of the present disclosure.
As shown in fig. 11, performing the image segmentation process on the sample image to obtain the segmentation labels of the sample image may include operations S1110 to S1130 as follows.
In operation S1110, an example segmentation process is performed on the sample image, resulting in an example segmentation label of the sample image.
In operation S1120, the tag is segmented according to the instance of the sample image, resulting in a semantic segmentation tag of the sample image.
In operation S1130, the semantic segmentation label of the sample image is used as the segmentation label of the sample image.
According to the embodiment of the disclosure, the instance is segmented into a plurality of instances which may belong to the same category and need to be distinguished. For example, for an object image, the object image may include a plurality of persons belonging to the category of persons, that is, a plurality of persons, and in the case segmentation, the plurality of persons need to be distinguished, and each person may obtain a corresponding case segmentation label.
According to the embodiment of the disclosure, semantic segmentation is to classify each pixel point in an image, but not to distinguish an instance. For example, for a target image, the target image may include a plurality of persons belonging to the category of persons, that is, a plurality of persons, and in semantic segmentation, the plurality of persons do not need to be distinguished, and the plurality of persons obtain the same semantic segmentation label.
According to an embodiment of the present disclosure, in order to make a more accurate semantic segmentation label, a Mask _ RCNN (cyclic convolution Neural Network) Network may be used to output instance segmentation labels on the three depth-estimated sample image databases CAD _60, CAD _120, and EPFL, and then the instance segmentation labels are converted into semantic segmentation labels, which are used as semantic segmentation labels on the three depth-estimated sample image databases. But is not limited thereto and the semantic segmentation tag may be used alone. Since the embodiment of the present disclosure employs a sample image database for depth estimation, a depth label of a sample image can be obtained.
According to other embodiments of the present disclosure, the Mask _ RCNN may detect and segment 82 types (including background types) of objects, in practical applications, the number of types appearing in the sample image database for depth estimation may be less than 82, and when the segmentation process is performed, the 82 types are directly used as segmentation labels, which will expand the segmentation range, resulting in increasing the error probability of the segmentation process.
According to the embodiment of the present disclosure, 59 categories appeared in the sample image database were used in order to construct dense segmentation labels. As shown in table 1, a mapping may be performed on the category of Mask _ RCNN, and the category that is not involved is marked as-1, so as to reduce the error probability and improve the segmentation effect and accuracy on the basis of the image segmentation processing.
TABLE 1
According to the embodiment of the disclosure, the result precision of the example segmentation label obtained by using the Mask _ RCNN network is high, and the construction of an image processing model is facilitated.
According to the embodiment of the disclosure, semantic segmentation can also be directly performed on a sample image database with depth labels, so as to obtain semantic segmentation labels of the sample images. In addition, the method is characterized in that the sample image is subjected to image segmentation processing on the basis of the sample image database with the depth label to obtain the segmentation label of the sample image, so that the sample image has the depth label and the segmentation label at the same time. The sample image can be subjected to depth estimation by adopting a segmentation label of a known sample image to obtain a depth label of the sample image, so that the sample image has the depth label and the segmentation label at the same time. However, the predicted depth map of the target object in the target image obtained by the first method has higher accuracy.
The following further describes the technical solution of the present disclosure with reference to a specific embodiment, and the operation of the image processing method may be specifically as follows.
FIG. 12 schematically shows a flow chart of yet another image processing method according to an embodiment of the present disclosure.
According to an embodiment of the present disclosure, as shown in fig. 12, the operation of the image processing method may include.
And (4) making a data set, constructing a full convolution neural network model, and training to obtain an image processing model.
Normalizing the input image to obtain an RGB image, and respectively obtaining a prediction depth map and a prediction segmentation map of the target image as the input of the image processing model.
And judging whether the corresponding prediction division map in the prediction division map of the target image is the target object, if so, applying the prediction division map of the target object to the prediction depth map of the target image to obtain the prediction depth map of the target object.
Furthermore, according to an embodiment of the present disclosure, it may also be determined whether to traverse the predicted partition maps of all target objects. If not, traversing the prediction segmentation graph of the target object of the next category; and if so, outputting the predicted depth maps of all the target objects.
According to the embodiment of the disclosure, the image processing method can be applied to an application scene in which the monocular robot searches or avoids a specific object, wherein the characteristic object is a target object.
Fig. 13 schematically shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.
As shown in fig. 13, the image processing apparatus 1300 includes an acquisition module 1310, a first processing module 1320, a determination module 1330, and a second processing module 1340.
The obtaining module 1310, the first processing module 1320, the determining module 1330, and the second processing module 1340 are communicatively coupled.
An obtaining module 1310 configured to obtain a target image, wherein the target image includes a target object and a non-target object.
The first processing module 1320 is configured to perform image segmentation processing and depth estimation processing on the target image, so as to obtain a predicted segmentation map and a predicted depth map of the target image respectively.
A determining module 1330 configured to determine a position of the target object in the predicted depth map of the target image according to the predicted segmentation map of the target object.
The second processing module 1340 is configured to process the predicted depth map according to the position of the target object in the predicted depth map of the target object, so as to obtain the predicted depth map of the target object.
According to the technical scheme of the embodiment of the disclosure, the target image is obtained and comprises a target object and a non-target object, the target image is subjected to image segmentation processing and depth estimation processing to respectively obtain a predicted segmentation map and a predicted depth map of the target image, the position of the target object in the predicted depth map of the target image is determined according to the predicted segmentation map of the target object, and the predicted depth map is processed according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object. The image segmentation and the depth estimation are combined, wherein the position of the target object in the predicted depth map can be obtained according to the predicted segmentation map, and the predicted depth map of the target object can be obtained by processing the predicted depth map according to the position of the target object in the predicted depth map, so that the technical problem that the depth estimation of the target object in the target image is difficult to realize in the related art is at least partially overcome, the depth of the target object in the target image is accurately determined, and the generalization of the method is strong.
The first processing module 1320 includes a first processing unit according to an embodiment of the present disclosure.
The image processing method comprises a first processing unit, a second processing unit and a third processing unit, wherein the first processing unit is used for processing a target image by using an image processing model to respectively obtain a segmentation map and a depth map, the image processing model is obtained by using a training sample for training, and the training sample comprises a sample image and a depth label and a segmentation label of the sample image.
According to an embodiment of the present disclosure, an image processing model includes a feature extraction network, an image segmentation network, and a depth estimation network.
According to an embodiment of the present disclosure, the first processing unit includes a first processing subunit, a second processing subunit, a third processing subunit, a fourth processing subunit, a fifth processing subunit, and a sixth processing subunit.
And the first processing subunit is used for processing the target image by using the feature extraction network to obtain a first intermediate feature map.
And the second processing subunit is used for processing the first intermediate feature map by using the image segmentation network to obtain a second intermediate feature map.
And the third processing subunit is used for processing the first intermediate feature map by using the depth estimation network to obtain a third intermediate feature map.
And the fourth processing subunit is used for generating a fourth intermediate feature map according to the second intermediate feature map and the third intermediate feature map.
And the fifth processing subunit is used for processing the fourth intermediate feature map by using the depth estimation network to obtain a predicted depth map of the target image.
And the sixth processing subunit is used for processing the second intermediate characteristic map by using the image segmentation network to obtain a predicted segmentation map of the target image.
According to an embodiment of the present disclosure, the image processing model is trained by using training samples, and may include the following operations.
Training samples are obtained. And training the full convolution neural network model by using the training sample to obtain an image processing model.
According to an embodiment of the present disclosure, a full convolution neural network model includes an initial feature extraction network, an initial image segmentation network, and an initial depth estimation network.
Training the full convolution neural network model by using the training samples to obtain an image processing model, which may include the following operations.
And utilizing the initial feature extraction network to process the sample image to obtain a fifth intermediate feature map. And processing the fifth intermediate feature map by using the initial image segmentation network to obtain a sixth intermediate feature map. And processing the fifth intermediate feature map by using the initial depth estimation network to obtain a seventh intermediate feature map. And generating an eighth intermediate feature map according to the sixth intermediate feature map and the seventh intermediate feature map. And processing the eighth intermediate feature map by using the initial depth estimation network to obtain a predicted depth map of the sample image. And processing the sixth intermediate characteristic graph by using the initial image segmentation network to obtain a prediction segmentation graph of the sample image. And inputting the depth label, the predicted depth map, the segmentation label and the predicted segmentation map of the sample image into a loss function of the full convolution neural network model to obtain a loss result. And adjusting the network parameters of the full convolution neural network model according to the loss result until the loss function is converged. And taking the trained full convolution neural network model as an image processing model.
According to an embodiment of the present disclosure, performing an image segmentation process on the sample image to obtain a segmentation label of the sample image may include the following operations.
And carrying out example segmentation processing on the sample image to obtain an example segmentation label of the sample image. And segmenting the label according to the sample image example to obtain the semantic segmentation label of the sample image. And taking the semantic segmentation label of the sample image as the segmentation label of the sample image.
The second processing module 1320 includes a second processing unit according to an embodiment of the present disclosure.
And the second processing unit is used for setting the pixel values of other positions except the position of the target image in the predicted depth map of the target image as preset pixel values to obtain the predicted depth map of the target object.
According to an embodiment of the present disclosure, the image segmentation process includes a semantic segmentation process or an instance segmentation process.
Any of the modules, units, or at least part of the functionality of any of them according to embodiments of the present disclosure may be implemented in one module. Any one or more of the modules and units according to the embodiments of the present disclosure may be implemented by being split into a plurality of modules. Any one or more of the modules, units according to the embodiments of the present disclosure may be implemented at least partially as a hardware Circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a Circuit, or implemented by any one of three implementations of software, hardware, and firmware, or any suitable combination of any of them. Alternatively, one or more of the modules, units according to embodiments of the present disclosure may be implemented at least partly as computer program modules, which, when executed, may perform the respective functions.
For example, any number of the obtaining module 1310, the first processing module 1320, the determining module 1330, and the second processing module 1340 may be combined and implemented in one module/unit, or any one of them may be split into a plurality of modules/units. Alternatively, at least part of the functionality of one or more of these modules/units may be combined with at least part of the functionality of other modules/units and implemented in one module/unit. According to an embodiment of the disclosure, at least one of the obtaining module 1310, the first processing module 1320, the determining module 1330, and the second processing module 1340 may be implemented at least partially as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in any one of three implementations of software, hardware, and firmware, or in a suitable combination of any of them. Alternatively, at least one of the obtaining module 1310, the first processing module 1320, the determining module 1330 and the second processing module 1340 may be at least partially implemented as a computer program module that, when executed, may perform a corresponding function.
It should be noted that the image processing apparatus portion in the embodiments of the present disclosure corresponds to the image processing method portion in the embodiments of the present disclosure, and the description of the image processing apparatus portion specifically refers to the image processing method portion, and is not repeated herein.
Fig. 14 schematically shows a block diagram of an electronic device adapted to implement the above described method according to an embodiment of the present disclosure. The electronic device shown in fig. 14 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 14, an electronic device 1400 according to an embodiment of the present disclosure includes a processor 1401, which can perform various appropriate actions and processes according to a program stored in a Read-Only Memory (ROM) 1402 or a program loaded from a storage portion 1408 into a Random Access Memory (RAM) 1403. Processor 1401 may comprise, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or associated chipset, and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), among others. The processor 1401 may also include onboard memory for caching purposes. Processor 1401 may include a single processing unit or multiple processing units for performing different actions of a method flow according to embodiments of the present disclosure.
In the RAM 1403, various programs and data necessary for the operation of the electronic device 1400 are stored. The processor 1401, the ROM 1402, and the RAM 1403 are connected to each other by a bus 1404. The processor 1401 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 1402 and/or the RAM 1403. Note that the programs may also be stored in one or more memories other than ROM 1402 and RAM 1403. The processor 1401 may also perform various operations of the method flows according to the embodiments of the present disclosure by executing programs stored in the one or more memories.
According to an embodiment of the present disclosure, electronic device 1400 may also include an input/output (I/O) interface 1405, which input/output (I/O) interface 1405 is also connected to bus 1404. The system 1400 may also include one or more of the following components connected to the I/O interface 1405: an input portion 1406 including a keyboard, a mouse, and the like; an output portion 1407 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker and the like; a storage portion 1408 including a hard disk and the like; and a communication portion 1409 including a network interface card such as a LAN card, a modem, or the like. The communication section 1409 performs communication processing via a network such as the internet. The driver 1410 is also connected to the I/O interface 1405 as necessary. A removable medium 1411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is mounted on the drive 1410 as necessary, so that a computer program read out therefrom is installed into the storage section 1408 as necessary.
According to embodiments of the present disclosure, method flows according to embodiments of the present disclosure may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable storage medium, the computer program containing program code for performing the method illustrated by the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 1409 and/or installed from the removable medium 1411. The computer program, when executed by the processor 1401, performs the above-described functions defined in the system of the embodiment of the present disclosure. The systems, devices, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
The present disclosure also provides a computer-readable storage medium, which may be contained in the apparatus/device/system described in the above embodiments; or may exist separately and not be assembled into the device/apparatus/system. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.
According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples may include, but are not limited to: a portable Computer diskette, a hard disk, a Random Access Memory (RAM), a Read-Only Memory (ROM), an Erasable Programmable Read-Only Memory (EPROM) or flash Memory), a portable compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the preceding. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
For example, according to embodiments of the present disclosure, a computer-readable storage medium may include one or more memories other than ROM 1402 and/or RAM 1403 and/or ROM 1402 and RAM 1403 described above.
According to the technical scheme of the embodiment of the disclosure, the target image is obtained and comprises a target object and a non-target object, the target image is subjected to image segmentation processing and depth estimation processing to respectively obtain a predicted segmentation map and a predicted depth map of the target image, the position of the target object in the predicted depth map of the target image is determined according to the predicted segmentation map of the target object, and the predicted depth map is processed according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object. The image segmentation and the depth estimation are combined, wherein the position of the target object in the predicted depth map can be obtained according to the predicted segmentation map, and the predicted depth map of the target object can be obtained by processing the predicted depth map according to the position of the target object in the predicted depth map, so that the technical problem that the depth estimation of the target object in the target image is difficult to realize in the related art is at least partially overcome, the depth of the target object in the target image is accurately determined, and the generalization of the method is strong.
Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the method provided by the embodiments of the present disclosure, when the computer program product is run on an electronic device, the program code being adapted to cause the electronic device to carry out the image processing method provided by the embodiments of the present disclosure.
The computer program, when executed by the processor 1401, performs the above-described functions defined in the system/apparatus of the embodiments of the present disclosure. The systems, apparatuses, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the present disclosure.
In one embodiment, the computer program may be hosted on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed in the form of signals over a network medium, downloaded and installed via the communication portion 1409, and/or installed from the removable media 1411. The computer program containing program code may be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.
In accordance with embodiments of the present disclosure, program code for executing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, these computer programs may be implemented using high level procedural and/or object oriented programming languages, and/or assembly/machine languages. The programming language includes, but is not limited to, programming languages such as Java, C + +, python, the "C" language, or the like. The program code may execute entirely on the user computing device, partly on the user device, partly on a remote computing device, or entirely on the remote computing device or server. In situations involving remote computing devices, the remote computing devices may be connected to the user computing device through any kind of Network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to external computing devices (e.g., through the internet using an internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. Those skilled in the art will appreciate that various combinations and/or combinations of features recited in the various embodiments and/or claims of the present disclosure can be made, even if such combinations or combinations are not expressly recited in the present disclosure. In particular, various combinations and/or combinations of the features recited in the various embodiments and/or claims of the present disclosure may be made without departing from the spirit or teaching of the present disclosure. All such combinations and/or associations are within the scope of the present disclosure.
The embodiments of the present disclosure have been described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described separately above, this does not mean that the measures in the embodiments cannot be used in advantageous combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be devised by those skilled in the art without departing from the scope of the present disclosure, and such alternatives and modifications are intended to be within the scope of the present disclosure.
Claims (12)
1. An image processing method comprising:
acquiring a target image, wherein the target image comprises a target object and a non-target object;
carrying out image segmentation processing and depth estimation processing on the target image to respectively obtain a predicted segmentation map and a predicted depth map of the target image;
determining the position of the target object in a predicted depth map of the target image according to the predicted segmentation map of the target object; and
and processing the predicted depth map according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object.
2. The method according to claim 1, wherein the performing image segmentation processing and depth estimation processing on the target image to obtain a predicted segmentation map and a predicted depth map of the target image respectively comprises:
processing the target image by using an image processing model to respectively obtain a prediction segmentation map and a prediction depth map of the target image, wherein the image processing model is obtained by training a training sample, and the training sample comprises a sample image and a depth label and a segmentation label of the sample image.
3. The method of claim 2, wherein the image processing model comprises a feature extraction network, an image segmentation network, and a depth estimation network;
the processing the target image by using the image processing model to respectively obtain the predicted segmentation map and the predicted depth map of the target image comprises the following steps:
processing the target image by using the feature extraction network to obtain a first intermediate feature map;
processing the first intermediate feature map by using the image segmentation network to obtain a second intermediate feature map;
processing the first intermediate feature map by using the depth estimation network to obtain a third intermediate feature map;
generating a fourth intermediate feature map according to the second intermediate feature map and the third intermediate feature map;
processing the fourth intermediate feature map by using the depth estimation network to obtain a predicted depth map of the target image; and
and processing the second intermediate characteristic graph by using the image segmentation network to obtain a prediction segmentation graph of the target image.
4. The method of claim 2, wherein the image processing model is trained using training samples, comprising:
obtaining the training sample; and
and training a full convolution neural network model by using the training sample to obtain the image processing model.
5. The method of claim 4, wherein the fully convolutional neural network model comprises an initial feature extraction network, an initial image segmentation network, and an initial depth estimation network;
training a full convolution neural network model by using the training sample to obtain the image processing model, wherein the training sample comprises the following steps:
processing the sample image by using the initial feature extraction network to obtain a fifth intermediate feature map;
processing the fifth intermediate feature map by using the initial image segmentation network to obtain a sixth intermediate feature map;
processing the fifth intermediate feature map by using the initial depth estimation network to obtain a seventh intermediate feature map;
generating an eighth intermediate feature map according to the sixth intermediate feature map and the seventh intermediate feature map;
processing the eighth intermediate feature map by using the initial depth estimation network to obtain a predicted depth map of the sample image;
processing the sixth intermediate feature map by using the initial image segmentation network to obtain a prediction segmentation map of the sample image;
inputting the depth label, the predicted depth map, the segmentation label and the predicted segmentation map of the sample image into a loss function of the full convolution neural network model, and adjusting network parameters of the full convolution neural network model according to the loss result until the loss function is converged; and
and taking the trained full convolution neural network model as the image processing model.
6. The method of claim 5, wherein the performing image segmentation processing on the sample image to obtain a segmentation label of the sample image comprises:
carrying out example segmentation processing on the sample image to obtain an example segmentation label of the sample image;
obtaining a semantic segmentation label of the sample image according to the example segmentation label of the sample image; and
and taking the semantic segmentation label of the sample image as the segmentation label of the sample image.
7. The method of claim 1, wherein the processing the predicted depth map according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object comprises:
and setting the pixel values of other positions except the position of the target image in the predicted depth map of the target image as preset pixel values to obtain the predicted depth map of the target object.
8. The method of any of claims 1-7, wherein the image segmentation process comprises a semantic segmentation process or an instance segmentation process.
9. An image processing apparatus comprising:
the device comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring a target image, and the target image comprises a target object and a non-target object;
the first processing module is used for carrying out image segmentation processing and depth estimation processing on the target image to respectively obtain a predicted segmentation map and a predicted depth map of the target image;
a determining module, configured to determine a position of the target object in a predicted depth map of the target image according to the predicted segmentation map of the target object; and
and the second processing module is used for processing the predicted depth map according to the position of the target object in the predicted depth map of the target image to obtain the predicted depth map of the target object.
10. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs,
wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-8.
11. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to carry out the method of any one of claims 1 to 8.
12. A computer program product comprising a computer program which, when executed by a processor, is adapted to carry out the method of any one of claims 1 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110002321.5A CN113781493A (en) | 2021-01-04 | 2021-01-04 | Image processing method, apparatus, electronic device, medium and computer program product |
| PCT/CN2021/140683 WO2022143366A1 (en) | 2021-01-04 | 2021-12-23 | Image processing method and apparatus, electronic device, medium, and computer program product |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110002321.5A CN113781493A (en) | 2021-01-04 | 2021-01-04 | Image processing method, apparatus, electronic device, medium and computer program product |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113781493A true CN113781493A (en) | 2021-12-10 |
Family
ID=78835376
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110002321.5A Pending CN113781493A (en) | 2021-01-04 | 2021-01-04 | Image processing method, apparatus, electronic device, medium and computer program product |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN113781493A (en) |
| WO (1) | WO2022143366A1 (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114170248A (en) * | 2021-12-21 | 2022-03-11 | 上海微创医疗机器人（集团）股份有限公司 | Image processing method, data processing method, medical system, device, and medium |
| WO2022143366A1 (en) * | 2021-01-04 | 2022-07-07 | 北京沃东天骏信息技术有限公司 | Image processing method and apparatus, electronic device, medium, and computer program product |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115527029A (en) * | 2022-09-14 | 2022-12-27 | 中国铁道科学研究院集团有限公司 | Image segmentation method based on deep learning model and computer equipment |
| CN115526926B (en) * | 2022-09-30 | 2026-02-17 | 驭势科技（北京）有限公司 | Training method of target depth estimation model and target depth estimation method |
| CN116029151B (en) * | 2023-02-09 | 2024-05-14 | 清华大学 | Water flow resistance prediction method, training method, flow prediction method and device |
| CN116342452A (en) * | 2023-03-29 | 2023-06-27 | 合肥国家实验室 | Image generation method and fusion imaging system |
| CN116563701B (en) * | 2023-04-19 | 2026-01-27 | 中国工商银行股份有限公司 | Target object detection method, device, equipment and storage medium |
| CN116597213B (en) * | 2023-05-18 | 2025-07-25 | 北京百度网讯科技有限公司 | Target detection method, training device, electronic equipment and storage medium |
| CN117437189B (en) * | 2023-10-17 | 2026-05-12 | 暨南大学 | Image auxiliary diagnosis method, device and storage medium based on instance segmentation |
| CN119888867B (en) * | 2025-03-27 | 2025-07-15 | 广东智云城建科技有限公司 | Gait recognition method fusing RGB image derived depth map and contour sequence |
| CN120451242B (en) * | 2025-05-09 | 2026-01-06 | 东方聚智(北京)科技创新有限公司 | Training Method and Apparatus for Visual Depth Estimation Model Based on Dynamic Uncertainty |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140362188A1 (en) * | 2013-06-07 | 2014-12-11 | Sony Computer Entertainment Inc. | Image processing device, image processing system, and image processing method |
| CN104346816A (en) * | 2014-10-11 | 2015-02-11 | 京东方科技集团股份有限公司 | Depth determining method and device and electronic equipment |
| CN106204522A (en) * | 2015-05-28 | 2016-12-07 | 奥多比公司 | The combined depth of single image is estimated and semantic tagger |
| CN109658413A (en) * | 2018-12-12 | 2019-04-19 | 深圳前海达闼云端智能科技有限公司 | A kind of method of robot target grasping body position detection |
| CN109785345A (en) * | 2019-01-25 | 2019-05-21 | 中电健康云科技有限公司 | Image segmentation method and device |
| CN110310229A (en) * | 2019-06-28 | 2019-10-08 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, terminal device and readable storage medium |
| CN110782468A (en) * | 2019-10-25 | 2020-02-11 | 北京达佳互联信息技术有限公司 | Training method and device of image segmentation model and image segmentation method and device |
| CN110969173A (en) * | 2018-09-28 | 2020-04-07 | 杭州海康威视数字技术股份有限公司 | Target classification method and device |
| US20200175700A1 (en) * | 2018-11-29 | 2020-06-04 | Adobe Inc. | Joint Training Technique for Depth Map Generation |
| CN111311560A (en) * | 2020-02-10 | 2020-06-19 | 中国铁道科学研究院集团有限公司基础设施检测研究所 | Method and device for detecting the state of rail fasteners |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| MY150361A (en) * | 2010-12-03 | 2013-12-31 | Mimos Berhad | Method of image segmentation using intensity and depth information |
| CN111968129B (en) * | 2020-07-15 | 2023-11-07 | 上海交通大学 | Semantic-aware real-time positioning and map construction system and method |
| CN113781493A (en) * | 2021-01-04 | 2021-12-10 | 北京沃东天骏信息技术有限公司 | Image processing method, apparatus, electronic device, medium and computer program product |
-
2021
- 2021-01-04 CN CN202110002321.5A patent/CN113781493A/en active Pending
- 2021-12-23 WO PCT/CN2021/140683 patent/WO2022143366A1/en not_active Ceased
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140362188A1 (en) * | 2013-06-07 | 2014-12-11 | Sony Computer Entertainment Inc. | Image processing device, image processing system, and image processing method |
| CN104346816A (en) * | 2014-10-11 | 2015-02-11 | 京东方科技集团股份有限公司 | Depth determining method and device and electronic equipment |
| CN106204522A (en) * | 2015-05-28 | 2016-12-07 | 奥多比公司 | The combined depth of single image is estimated and semantic tagger |
| CN110969173A (en) * | 2018-09-28 | 2020-04-07 | 杭州海康威视数字技术股份有限公司 | Target classification method and device |
| US20200175700A1 (en) * | 2018-11-29 | 2020-06-04 | Adobe Inc. | Joint Training Technique for Depth Map Generation |
| CN109658413A (en) * | 2018-12-12 | 2019-04-19 | 深圳前海达闼云端智能科技有限公司 | A kind of method of robot target grasping body position detection |
| CN109785345A (en) * | 2019-01-25 | 2019-05-21 | 中电健康云科技有限公司 | Image segmentation method and device |
| CN110310229A (en) * | 2019-06-28 | 2019-10-08 | Oppo广东移动通信有限公司 | Image processing method, image processing apparatus, terminal device and readable storage medium |
| CN110782468A (en) * | 2019-10-25 | 2020-02-11 | 北京达佳互联信息技术有限公司 | Training method and device of image segmentation model and image segmentation method and device |
| CN111311560A (en) * | 2020-02-10 | 2020-06-19 | 中国铁道科学研究院集团有限公司基础设施检测研究所 | Method and device for detecting the state of rail fasteners |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022143366A1 (en) * | 2021-01-04 | 2022-07-07 | 北京沃东天骏信息技术有限公司 | Image processing method and apparatus, electronic device, medium, and computer program product |
| CN114170248A (en) * | 2021-12-21 | 2022-03-11 | 上海微创医疗机器人（集团）股份有限公司 | Image processing method, data processing method, medical system, device, and medium |
| CN114170248B (en) * | 2021-12-21 | 2025-03-25 | 上海微创医疗机器人（集团）股份有限公司 | Image processing method, data processing method, medical system, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022143366A1 (en) | 2022-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12061989B2 (en) | Machine learning artificial intelligence system for identifying vehicles | |
| WO2022143366A1 (en) | Image processing method and apparatus, electronic device, medium, and computer program product | |
| US11392792B2 (en) | Method and apparatus for generating vehicle damage information | |
| US11205100B2 (en) | Edge-based adaptive machine learning for object recognition | |
| US20200117906A1 (en) | Space-time memory network for locating target object in video content | |
| CN108256479B (en) | Face tracking method and device | |
| WO2022105125A1 (en) | Image segmentation method and apparatus, computer device, and storage medium | |
| CN113704531A (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| US12620197B2 (en) | Image processing method, apparatus, computing device, and medium | |
| CN109118456B (en) | Image processing method and device | |
| CN113569740B (en) | Video recognition model training method and device, and video recognition method and device | |
| AU2016201908A1 (en) | Joint depth estimation and semantic labeling of a single image | |
| US20250022100A1 (en) | Multi-modal synthetic content generation using neural networks | |
| Li et al. | VNLSTM-PoseNet: A novel deep ConvNet for real-time 6-DOF camera relocalization in urban streets | |
| US12488221B2 (en) | Unsupervised pre-training of neural networks using generative models | |
| CN112329762A (en) | Image processing method, model training method, apparatus, computer equipment and medium | |
| CN112861940A (en) | Binocular disparity estimation method, model training method and related equipment | |
| CN111292333A (en) | Method and apparatus for segmenting an image | |
| US20180165539A1 (en) | Visual-saliency driven scene description | |
| US12282987B2 (en) | Generating image mattes without trimap segmentations via a multi-branch neural network | |
| US12008792B1 (en) | Independently determining adjustments to bounding shapes for detected objects in image data | |
| CN112651351B (en) | Data processing method and device | |
| CN116758554A (en) | A method, device, electronic device and storage medium for extracting test question information | |
| CN115482279A (en) | Object pose estimation method, device, medium and equipment | |
| CN115713769A (en) | Training method and device of text detection model, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20211210 |
|
| RJ01 | Rejection of invention patent application after publication |