WO2021093448A1 - 图像处理方法、装置、服务器、医疗图像处理设备及存储介质 - Google Patents

图像处理方法、装置、服务器、医疗图像处理设备及存储介质 Download PDF

Info

Publication number
WO2021093448A1
WO2021093448A1 PCT/CN2020/115768 CN2020115768W WO2021093448A1 WO 2021093448 A1 WO2021093448 A1 WO 2021093448A1 CN 2020115768 W CN2020115768 W CN 2020115768W WO 2021093448 A1 WO2021093448 A1 WO 2021093448A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
abnormal
positioning
target
processing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/115768
Other languages
English (en)
French (fr)
Inventor
罗莉舒
尚鸿
孙钟前
郑瀚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology (Shenzhen) Co Ltd
Original Assignee
Tencent Technology (Shenzhen) Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology (Shenzhen) Co Ltd filed Critical Tencent Technology (Shenzhen) Co Ltd
Priority to EP20886491.8A priority Critical patent/EP4006831A4/en
Publication of WO2021093448A1 publication Critical patent/WO2021093448A1/zh
Priority to US17/515,174 priority patent/US12051199B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/0895—Weakly supervised learning, e.g. semi-supervised or self-supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10068—Endoscopic image
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30092—Stomach; Gastric
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30096—Tumor; Lesion

Definitions

  • This application relates to the image processing field of artificial intelligence technology, in particular to image processing methods, devices, servers, medical image processing equipment and storage media.
  • analyzing the image including the abnormal area to determine the abnormality category to which the abnormal area belongs and the specific location where the abnormality occurs is a commonly used method of abnormality classification.
  • doctors can determine the patient’s disease and the location of the disease by analyzing and processing the images; in the field of vehicle detection, maintenance personnel or other
  • the technician can determine the type of failure described in the vehicle failure, and the specific location where the failure occurred or caused the failure.
  • the embodiments of the present application provide an image processing method, device, server, medical image processing equipment, and storage medium, which can perform abnormality detection on images to be detected more efficiently.
  • an embodiment of the present application provides an image processing method, which is executed by a server, and includes:
  • the initial positioning image and the target positioning image are used to reflect the attribute information of the target area associated with the abnormal category prediction tag.
  • an image processing device including:
  • the acquiring unit is used to acquire the image to be detected
  • the processing unit is used to perform down-sampling abnormal classification processing on the image to be detected to obtain an abnormal category prediction label and a target feature image; perform preliminary abnormal location processing based on the abnormal category prediction label and the target feature image to obtain the image to be detected Corresponding initial positioning image; performing up-sampling abnormal positioning processing on the initial positioning image to obtain the target positioning image of the image to be detected;
  • An output unit configured to output the abnormal category prediction label and the target location image
  • the initial positioning image and the target positioning image are used to reflect the attribute information of the target area associated with the abnormal category prediction tag.
  • an embodiment of the present application provides a server, and the server includes:
  • a computer storage medium storing one or more instructions, and the one or more instructions are suitable for being loaded by the processor and executing the above-mentioned image processing method.
  • an embodiment of the present application provides a medical image processing device, the medical image collector, the medical image analyzer, and the medical image display, the medical image collector is connected to the medical image analyzer, so The medical image display is connected to the medical image analyzer;
  • the medical image collector is used to obtain medical images
  • the medical image analyzer is used to perform down-sampling disease classification processing on the medical image to obtain disease category prediction labels and target feature images; perform preliminary disease location processing based on the disease category prediction labels and target feature images to obtain all The initial positioning image corresponding to the medical image; the medical image analyzer is also used to perform up-sampling disease positioning processing on the initial positioning image to obtain a target positioning image corresponding to the medical image; predicting a label for the disease category And outputting the target positioning image to the medical image display;
  • the medical image display is used to display the disease category prediction label and the target location image
  • the initial positioning image and the target positioning image are used to reflect the attribute information of the target area associated with the disease category prediction tag.
  • an embodiment of the present application provides a computer storage medium that stores one or more instructions, and the one or more instructions are suitable for being loaded by a processor and executing the above-mentioned image processing method.
  • the embodiments of the present application also provide a computer program product or computer program.
  • the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium.
  • the processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above-mentioned image processing method.
  • Figure 1a is a schematic structural diagram of an image processing system provided by an embodiment of the present application.
  • Figure 1b is a schematic structural diagram of an abnormal location model provided by an embodiment of the present application.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of the present application.
  • Fig. 3 is a schematic diagram of determining a target feature weight vector provided by an embodiment of the present application.
  • Fig. 4a is a schematic diagram of an initial positioning image provided by an embodiment of the present application.
  • FIG. 4b is a schematic diagram of a target positioning image provided by an embodiment of the present application.
  • FIG. 5 is a schematic structural diagram of another image detection model provided by an embodiment of the present application.
  • FIG. 6 is a schematic flowchart of another image processing method provided by an embodiment of the present application.
  • Fig. 7a is a schematic diagram of a second sample image and an optimized image provided by an embodiment of the present application.
  • FIG. 7b is a schematic diagram of determining the first probability and the second probability according to an embodiment of the present application.
  • Fig. 8a is a schematic structural diagram of a medical image processing device provided by an embodiment of the present application.
  • Fig. 8b is a schematic diagram of image processing performed by a medical image processing device provided by an embodiment of the present application.
  • Fig. 8c is a schematic diagram of a user interface of a medical image display provided by an embodiment of the present application.
  • Fig. 9a is an application scenario diagram of medical image processing provided by an embodiment of the present application.
  • Fig. 9b is a schematic diagram of medical image processing provided by an embodiment of the present application.
  • FIG. 10 is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application.
  • Fig. 11 is a schematic structural diagram of a server provided by an embodiment of the present application.
  • the solution provided by the embodiments of this application relates to computer vision technology in artificial intelligence technology.
  • the computer vision (Computer Vision, CV) technology can use cameras and smart terminals instead of human eyes to identify, track, and measure target objects and process output changes. It is suitable for human eyes to observe or send images to be inspected by specific instruments.
  • the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
  • the image detection model used to detect abnormal categories and abnormal locations can be pre-trained.
  • the pre-trained image detection model is called to analyze the image to be detected.
  • the abnormal category included in the image to be detected and the attribute information of the abnormal area where the abnormality occurs can be obtained, and the attribute information may include the specific location and size of the abnormal area.
  • the meaning of the abnormal area is different.
  • the so-called abnormal area may refer to the area where a disease occurs; in the vehicle field, the abnormal area may refer to a malfunctioning vehicle component; in other fields, the abnormal area may refer to the area where a specific object is located.
  • the image to be detected includes clouds, the sun, and an airplane, and the location of the cloud, sun, or airplane can be referred to as an abnormal area of the image to be detected.
  • training the image detection model is based on conventional supervised learning training image detection model.
  • the so-called training image detection model based on conventional supervised learning means that the label corresponding to the training sample has the same granularity as the function of the image detection model.
  • this model requires manual labeling of anomaly category labels and anomaly location labels for the training samples, and the workload is large and cumbersome.
  • a training method based on weakly supervised learning is used to train the image detection model.
  • the so-called weakly supervised learning means that the granularity of the label corresponding to the training sample is lower than the granularity of the function of the image detection model.
  • the function of the image detection model is to classify and identify the disease in the endoscopic image and locate the diseased area where the disease occurs.
  • the training sample only has the corresponding disease category label.
  • this method eliminates the step of labeling positioning tags, can save manpower and material resources, and at the same time avoids the impact of positioning tag errors on the accuracy of the image detection model.
  • Fig. 1a is a schematic structural diagram of an image processing system provided by an embodiment of the present application.
  • the image processing system 100 includes a user 101, an image acquisition terminal 120, a server 103, and an image display device 104.
  • the server 103 includes a processor and a memory, and the method embodiments in this application are executed by the processor executing instructions stored in the memory.
  • the server 103 includes an image processing device 1031.
  • the image collection terminal 120 refers to any kind of intelligent terminal with an image collection function.
  • the image collection terminal 120 is specifically a medical collection device with a camera, for example, an endoscope device, which performs endoscopic detection on the user 101, and the video collection module configured on the endoscope device collects the endoscope.
  • Video data, and the collected frames of images are sent to the server 103.
  • the server 103 recognizes, tracks, and detects the target object.
  • the image processing device 1031 therein can process the image to be detected to obtain the abnormal category prediction label and the target location image, and send it to the image display device 104 for display.
  • the detection result is displayed to the doctor to learn whether the detection part of the user 101 is abnormal, and the part of the image where the lesion occurred when the abnormality occurred.
  • the server 110 may be a server, or a server cluster composed of several servers, or a cloud computing service center.
  • FIG. 1b it is a schematic structural diagram of an image detection model provided by an embodiment of this application.
  • the schematic structural diagram shown in Fig. 1b may include an abnormal classification sub-model 101, an initial positioning module 102, and an abnormal positioning sub-model 103.
  • the abnormal classification sub-model 101 is used to perform down-sampling abnormal classification processing
  • the initial positioning module 102 is used to perform down-sampling abnormalities.
  • the processing result performs preliminary abnormal positioning
  • the abnormal positioning sub-model 103 is used to perform up-sampling abnormal positioning processing based on the preliminary abnormal positioning result.
  • the anomaly classification sub-model 101 may be a trained model with anomaly classification capability, which may be directly used for down-sampling anomaly processing; or, the anomaly classification sub-model 101 may also be an untrained model. Since the initial positioning module 102 and the abnormal positioning sub-model 103 need to directly or indirectly rely on the processing results of the abnormal classification sub-model 101 when performing abnormal positioning, in order to ensure the accuracy of abnormal positioning, first ensure the abnormal classification sub-model 101 The result of down-sampling abnormal classification processing is more accurate. If the abnormal classification sub-model 101 is an untrained model, when training the image detection model, the abnormal classification sub-model 101 must be trained first.
  • the anomaly classification sub-model 101 may be any image recognition model with an image recognition function.
  • the convolutional neural network model has become a commonly used image recognition model due to its strong feature extraction performance.
  • the abnormal classification sub-model 101 may be a convolutional neural network model, such as a res-net model, Any one of dense-net model, VGG and GoogleNet model.
  • the convolutional neural network model mainly includes a fully convolutional layer responsible for feature extraction, a pooling layer for reducing the size of input data, and a fully connected layer for calculating the classification value of the classification based on the output of the pooling layer, And the softmax layer that further performs classification processing and outputs the classification results.
  • the pooling layer may include a global average pooling layer, a maximum pooling layer, an overlapping pooling layer, etc.
  • the pooling layer is different in different convolutional neural network models. Among them, each convolutional layer is responsible for extracting different features to obtain a feature image, and the feature image obtained by the previous convolutional layer is used as the input of the latter convolutional layer.
  • the features that each convolutional layer is responsible for extracting can be set according to a specific function, or set artificially.
  • the first convolutional layer can be set to be responsible for extracting the overall shape features of the graphics; the second convolutional layer is responsible for extracting the line features of the graphics; the third convolutional layer is responsible for extracting the discontinuities of the graphics Features etc.
  • the first convolutional layer when recognizing an image containing a human face, can be set to be responsible for extracting contour features of the human face; the second convolutional layer can be responsible for extracting facial features of the human face.
  • Each convolutional layer includes multiple feature extraction channels, and each feature extraction channel includes multiple filters of the same size for performing convolution calculations.
  • the anomaly classification sub-model 101 specifically includes: a fully convolutional layer (also called a down-sampling layer) and a softmax layer.
  • the full convolutional layer includes multiple convolutional layers.
  • the anomaly classification sub-model 101 further includes a global average pooling layer and a fully connected layer (not shown in FIG. 1b), and the global average pooling layer is located between the fully convolutional layer and the fully connected layer, The fully connected layer is connected to the softmax layer.
  • the abnormality classification sub-model 101 is trained based on the first sample image and the abnormal category supervision label corresponding to the first sample image, and the specific training process will be described in detail later.
  • the target positioning sub-model 103 is trained based on the trained anomaly classification sub-model 101 and the initial positioning module 102.
  • the specific process may be: acquiring a second sample image, which may be the same as or different from the first sample image; calling the trained anomaly classification sub-model 101 to perform anomaly classification processing on the second sample image to obtain Abnormal category training label; and extract the training feature image generated in the process of abnormal classification processing on the second sample image, the training feature image may be the last convolution layer of the full convolution layer of the abnormal classification sub-model 101 for feature extraction owned.
  • the initial positioning module 102 performs abnormal positioning processing based on the training feature image and the abnormal category training label, and obtains the first positioning image for abnormally positioning the abnormal area in the second sample image.
  • the initial positioning module 102 may include two parts: a feature filter and a category identification acquisition module.
  • the feature filter is used to process the training feature image
  • the category identification acquisition module is used to acquire the category identification corresponding to the abnormal category training label. It can be the identification number corresponding to the abnormal category indicated by the abnormal category training tag, for example, the identification number 0 corresponds to no abnormality, the identification number 1 corresponds to the abnormality type 1, and the identification number 2 corresponds to the abnormality type 2.
  • the abnormal positioning sub-model 103 calls the abnormal positioning sub-model 103 to perform abnormal positioning processing on the first positioning image to obtain the second positioning image; finally, determine the target loss function for training the abnormal positioning sub-model 103 according to the second positioning image and the second sample image
  • the abnormal location sub-model 103 is optimized according to the direction of reducing the value of the objective loss function, and the trained abnormal location sub-model is finally obtained.
  • the training of the abnormal location sub-model 103 is completed, which means that the training of the image detection model is completed.
  • each sub-model or module in the image detection model after the training can be called to process the image to be detected to obtain the abnormality category included in the image to be detected, the abnormal location and the size of the abnormal area where the abnormality occurs.
  • FIG. 2 is a schematic flowchart of an image processing method provided by an embodiment of this application.
  • the image processing method shown in FIG. 2 may be executed by a server, and specifically may be executed by a processor of the server, for example, the server 103 shown in FIG. 1a.
  • the image processing method shown in FIG. 2 may include the following steps:
  • Step S201 Obtain an image to be detected, and perform down-sampling abnormal classification processing on the image to be detected to obtain an abnormal category prediction label and a target feature image.
  • the image to be detected may be sent by the terminal.
  • a doctor performs endoscopic detection on a patient through an endoscopic device
  • a video acquisition module configured on the endoscopic device collects endoscopic video data
  • the endoscope device or the terminal connected to the endoscope can perform image framing processing on the video data to obtain multi-frame images; further, the obtained multi-frame images are sent to the server as the images to be detected; in other embodiments, all The image to be detected may also be obtained by the server receiving the video data sent by the terminal and performing image framing processing on the video data.
  • the abnormal category prediction label is used to reflect whether there is an abnormality in the image to be detected, and if there is an abnormality, the abnormality type to which the abnormality belongs.
  • the abnormal category prediction tag may indicate whether there is a disease in the image to be detected, and if there is a disease, what type of disease the disease belongs to, such as tumor, gastric cancer, and liver cirrhosis.
  • the abnormal category prediction label may include a target category identifier used to indicate a certain abnormality, or the abnormal category prediction label may also include the probability of the existence of the abnormal category.
  • the target category identifier of no abnormality is preset as 0, the category identifier of abnormal category 1 is 1, and the category identifier of abnormal category 2 is 2.
  • the abnormal category prediction label may be (1, 60%), (2, 40%), where the numbers 1 and 2 identify the category identifier, and the following percentage indicates the probability of the corresponding abnormal category.
  • the number of the aforementioned abnormal category prediction label is one. If anomaly classification is performed on the image to be detected and multiple anomaly classification prediction labels are obtained, the one with the highest confidence is selected as the anomaly category prediction label corresponding to the image to be detected.
  • the purpose of performing down-sampling abnormal processing on the image to be detected is to determine whether there is an abnormality in the image to be detected, and if there is an abnormality, specifically what kind of abnormality is.
  • the manner of performing down-sampling abnormal processing on the image to be detected may include: performing convolution processing on the image to be detected by at least one convolution module to extract features included in the image to be detected; and based on the extracted features Perform anomaly detection to obtain anomaly category prediction labels.
  • the aforementioned at least one convolution module may be directly configured in the server.
  • the abnormal classification sub-model may be run in the server, and when the abnormal classification sub-model is used to perform down-sampling abnormal classification processing on the image to be detected, it specifically includes: obtaining the abnormal classification sub-model, where the abnormal classification sub-model includes at least one first sampling module Call at least one first sampling module, perform convolution processing on the image to be detected, and extract the features included in the image to be detected; perform anomaly detection based on the extracted features to obtain the abnormal category prediction label; The characteristic image output by the last sampling module in the at least one first sampling module is determined to be the target characteristic image.
  • the anomaly classification sub-model may include a convolutional layer, and the above-mentioned at least one adopting module is specifically a convolution module, which corresponds to the convolutional layer of the anomaly classification sub-model.
  • the convolutional layer in which the convolution module is the anomaly classification sub-model is taken as an example for description.
  • the abnormal classification sub-model is mainly used to perform abnormal classification on the image to be detected, and obtain the abnormal category prediction label of the image to be detected.
  • the abnormal classification sub-model can be obtained by training based on the first sample image and the abnormal category supervision label corresponding to the first sample image.
  • the abnormality category supervision label reflects whether an abnormality is included in the first sample image and if an abnormality is included, the abnormality category to which the abnormality belongs.
  • the abnormal category supervision label may be obtained by manually labeling the first sample image; or, the abnormal category supervision label may also be obtained by manual labeling and a labeling model.
  • Each convolutional layer of the abnormal classification sub-model includes multiple feature extraction channels, and each feature extraction channel includes multiple filters with the same size for convolution calculation.
  • the manner of performing down-sampling abnormal processing on the image to be detected may include: performing convolution processing on the image to be detected through at least one convolution module, including: processing the image through multiple filters in at least one convolution module.
  • the detection image is subjected to convolution processing.
  • performing convolution processing on the image to be detected means that each convolution module performs convolution processing on the image to be detected to obtain the corresponding feature image; this feature map is used as the input of the next convolution module to make the next convolution The module performs convolution processing on the feature map, and so on, until the last convolution module performs convolution processing, and outputs the feature image of the last layer.
  • the feature image of the last layer contains the spatial structure of the image to be detected. information.
  • the target feature image refers to a feature image generated in the process of down-sampling anomaly classification processing on the image to be detected, in order to obtain accuracy when the subsequent preliminary abnormal location processing is performed based on the target feature image and the abnormal category prediction label
  • the feature image output by the last convolution module is determined as the target feature image.
  • Step S202 Perform preliminary abnormal positioning processing based on the abnormal category prediction label and the target feature image to obtain an initial positioning image corresponding to the image to be detected.
  • the initial positioning image may be used to roughly reflect the attribute information of the target area associated with the abnormal category prediction tag, and the attribute information includes the location and size of the target area.
  • the target area associated with the abnormal category prediction label if the abnormal category prediction label indicates that there is no abnormality in the image to be detected, the target area associated with the abnormal category prediction label does not exist; if the abnormal category prediction label indicates that there is a certain type of abnormality in the image to be detected, the The target area associated with the anomaly category prediction label refers to the area where this type of anomaly occurs.
  • the attribute information of the target area may be displayed in the form of a label box, where the position of the label box indicates the location of the area where the abnormality occurs, and the size of the label box represents the size of the area where the abnormality occurs.
  • the implementation of performing preliminary abnormal location processing based on the abnormal category prediction label and target feature image to obtain the initial location image corresponding to the image to be detected may be: predicting the target category identifier corresponding to the label and the target based on the abnormal category prediction label.
  • the characteristic image undergoes preliminary abnormal positioning processing to obtain the initial positioning image.
  • the target category identifier corresponding to the abnormal category prediction label is used to uniquely identify the abnormality indicated by the abnormal category prediction label.
  • the target category identifier may be an abnormal category number, and the abnormal category number may be preset, such as in medical In the field, it can be set to abnormal category number 0 to indicate no lesions, abnormal category number 1 to indicate gastritis, abnormal category number 2 to indicate enteritis, abnormal category number 3 to indicate gastric cancer and so on.
  • step S302 performing preliminary abnormal positioning processing based on the abnormal category prediction label and the target feature image to obtain the initial positioning image corresponding to the image to be detected includes: obtaining the abnormal category prediction The target category identification corresponding to the label; the initial positioning module is called to obtain the target feature weight vector corresponding to the target category identification, and the target feature weight vector and the target feature image are superimposed to obtain the initial positioning image .
  • the initial positioning module may be a module configured in the server, or a module that runs independently.
  • the initial positioning module may include a feature filter and a category identification acquisition module.
  • the initial positioning module may acquire a target category identification through the category identification acquisition module, and the feature filter is used to acquire a target feature weight vector corresponding to the target category identification.
  • the implementation manner for obtaining the target category identifier corresponding to the abnormal category prediction label may be: the server may preset the correspondence between multiple abnormal categories and category numbers, and when the image to be detected is down-sampled When the abnormality classification process obtains the abnormality category prediction label, the abnormality indicated by the abnormality category prediction label is determined, and further, the category identification acquisition module searches for the target category identification corresponding to the determined abnormality in the preset correspondence relationship.
  • the implementation manner of acquiring the target category identifier corresponding to the abnormal category prediction label may also be: the category identifier acquiring module receives the target category identifier sent by the terminal. It should be understood that after the abnormal category prediction label is determined in step S201, the abnormal category prediction label may be sent to the terminal, and the terminal displays the abnormal category prediction label on the user interface of the terminal for the user to view. If the user wants to further locate the abnormal location of the image to be detected, he can input the triggering operation of the location processing in the user interface of the terminal.
  • the user interface of the terminal may include a trigger option for abnormal location. When the user clicks on this option, it indicates that the user has input Trigger operation for positioning processing. The user can select or input the target category identification corresponding to the abnormal category prediction label in the user interface while inputting the trigger operation; when the terminal detects the trigger operation, the target category identification input by the user is sent to the category identification acquisition module.
  • the user when the user inputs a triggering operation for positioning processing in the user interface, the user should also indicate which abnormal category prediction tag represents the abnormality to be located.
  • the anomaly classification sub-model can also include a fully connected layer and a pooling layer.
  • the pooling layer is used to reduce the target feature map output by the last convolutional layer, such as the target feature image output by the last convolutional layer.
  • N represents the number of feature extraction channels
  • 2048 represents the number of filters in each feature extraction channel
  • 7x7 represents the size of the filter
  • the target feature image is globally averaged pooled
  • the characteristic image obtained afterwards is [N, 1, 1, 2048].
  • the fully connected layer is used to calculate the classification value based on the pooled target feature image.
  • the target feature weight vector corresponding to the target category identifier there are generally two methods: one is a non-training method, and the other is a training method.
  • a non-training method a method for determining a target feature weight vector and an initial positioning image through a non-training method provided by an embodiment of this application.
  • the input image to be detected is passed through the full convolutional layer to obtain the target feature image; then the target feature image is globally averaged pooled, and finally input to the fully connected layer to obtain the classification result.
  • the target feature image is pooled by global average
  • the target feature image of each channel after pooling has a corresponding weight parameter, which can be recorded as
  • x represents an abnormal category
  • k represents the target feature image of each channel after pooling
  • the value of k can be between 1 and N
  • N represents the number of channels.
  • the weight parameter associated with the abnormality indicated by the target category identifier is c
  • the weight parameter corresponding to the abnormal category c is selected as the target feature weight vector, which can be recorded as
  • the target feature weight vector corresponding to the abnormal category c is weighted with the target feature image before pooling to obtain a heat map, which is the initial positioning image.
  • a ReLU is added after the target feature weight vector and the target feature image are weighted.
  • the addition of ReLU can filter out other pixels that do not belong to the abnormal category c, which can enhance the influence on the abnormal category c Of pixels.
  • the fully connected layer uses the global average pooling fully connected layer instead of the traditional fully connected layer. This is because the traditional fully connected layer combines the multiple layers of the fully convolutional layer. The neuron is directly stretched into a vector, destroying the spatial structure information between pixels, and the amount of parameters is huge.
  • the global average pooling method can retain the spatial information contained in the target feature image, which is beneficial to locate the target abnormal area.
  • the essence of embedding is also to learn a mapping relationship, that is, the mapping from the semantic space (category number) to the vector space (the weight vector of each channel of the feature map). For example, suppose the total number of abnormal categories is m, and the embedding parameter is The size is [m, N].
  • the target category identification is a vector selector (for example, the target category identification 0: [0])
  • the size of the vector embedding_vec obtained after the embedding operation is [1, N], that is, the abnormal category 0 corresponds to the vector mapped to the N-length vector space , Input different category identifiers, and after embedding operation, different vectors in N-length vector space can be obtained.
  • the weight vector on the graph Finally, activation_vec*F_n(x,y) is weighted to get the preliminary positioning image.
  • Step S203 Up-sampling abnormal positioning processing is performed on the initial positioning image to obtain a target positioning image corresponding to the image to be detected.
  • the target positioning image can be used to more accurately reflect the attribute information of the target area associated with the abnormal category prediction tag.
  • FIG. 4a is a schematic diagram of an initial positioning image provided by an embodiment of this application
  • FIG. 4b is a schematic diagram of a target positioning image provided by an embodiment of this application.
  • 401 represents the target area
  • 402 in FIG. 4b represents the target.
  • the area as can be seen from the figure, in Figure 4a, in addition to the target area, you can also see other areas 403; however, in Figure 4b you can clearly see the location and size of the target area, and you can’t see other areas .
  • the implementation manner of performing up-sampling abnormal positioning processing on the initial positioning image to obtain the target positioning image may be: obtaining an abnormal positioning sub-model, where the abnormal positioning sub-model includes at least one second sampling module, and The number of first sampling modules included in the abnormal classification sub-model is greater than the number of second sampling modules; at least one second sampling module is called to perform up-sampling abnormal positioning processing on the initial positioning image to obtain the target positioning image.
  • the specific method of obtaining the abnormal location sub-model includes: obtaining the second sample image; calling the abnormal classification sub-model, and performing down-sampling abnormal classification processing on the second sample image to obtain the abnormal category training label;
  • the initial positioning module performs preliminary abnormal positioning processing based on the abnormal category training tag to obtain a first positioning image; trains to obtain an abnormal positioning sub-model based on the abnormal category training tag and the first positioning image.
  • the above-mentioned abnormal classification sub-model, initial positioning module, and abnormal positioning sub-model may form an image detection model, and the image detection model may be used to detect the image to be detected and output the abnormal category prediction corresponding to the image to be detected.
  • the abnormal classification sub-model, the initial positioning module, and the abnormal positioning sub-model may also be three independent models, and the server may call the abnormal classification sub-model to perform step S201, call the initial positioning model to perform step 202, and call the abnormal positioning The sub-model executes step S203.
  • an image detection model composed of an abnormal classification sub-model, an initial positioning module, and an abnormal positioning sub-model is taken as an example.
  • a schematic diagram of the image detection model may be shown in FIG. 1b.
  • Step S204 Output the abnormal category prediction label and the target location image.
  • the server may output the abnormal category prediction label and the target location image to the terminal together, and the terminal displays it on the user interface for the user to view.
  • step S201 the abnormal classification sub-model in the image detection model shown in FIG. 1b is called to perform down-sampling the abnormal classification processing on the image to be detected to obtain the abnormal category prediction label and the target feature image.
  • the down-sampling Anomaly classification processing includes multiple convolution modules of an anomaly classification sub-model to perform convolution processing on the image to be detected, and the target feature image may be obtained by convolution processing by the last convolution module.
  • step S202 the resolution of the initial positioning image obtained based on the target feature map and the abnormal category prediction label is low, and the initial positioning image needs to be further up-sampled and refined through step S203 to obtain a target positioning image with higher resolution.
  • the number of convolution modules used for downsampling processing in the anomaly classification sub-model should be greater than the number of upsampling processing used for upsampling in the anomaly location sub-model The number of modules.
  • 501 represents an abnormal classification sub-model
  • 502 represents an initial positioning module
  • 503 represents an abnormal localization sub-model.
  • the anomaly classification sub-model 501 is the resent-50 model in the convolutional neural network model.
  • the resent-50 model includes 5 convolution modules and a softmax layer.
  • the size of the image to be detected is 224*224, after 5 convolution modules After 5 times of downsampling, the size of the target feature map is 7*7.
  • the initial positioning module performs initial positioning processing on the target feature image to obtain the initial positioning image and the size remains unchanged at 7*7; further, the abnormal positioning sub-model 503 performs three upsampling processing on the initial positioning image, and the obtained target positioning image is The size is 56*56, the side length is only 1/4 of the image to be detected, and then the size of the target positioning image is restored to 224*224 through interpolation.
  • the reason why five up-sampling modules are not directly used in the target location sub-module is because there is a shortcut between the abnormal classification sub-model 501 and the abnormal location sub-model 503 in the image detection model shown in FIG. Detect images to facilitate feature alignment.
  • the higher-level feature images include lower-level semantics, and the abnormal location sub-model 503 needs more high-level semantic information with judgment when positioning, and too much lower-level semantics will make the positioning inaccurate.
  • the abnormal category prediction label can be obtained by down-sampling the abnormality processing of the image to be detected, which saves manpower and is more efficient compared with the method of artificially analyzing the abnormality in the traditional technology; further, the embodiment of the present application
  • the initial abnormal location processing is performed, and then the initial location image obtained from the preliminary abnormal processing is up-sampled abnormal location processing to obtain the image to be detected
  • the corresponding target positioning image after two positioning processing, can ensure that the obtained target positioning image is more accurate.
  • an embodiment of the present application also provides another image processing method.
  • FIG. 6 is a flowchart of another image processing method provided by an embodiment of the present application.
  • the image processing method shown in FIG. 6 can be executed by a server and a server calling an image detection model.
  • the server 103 shown in FIG. 1a may include an abnormal classification sub-model, an initial positioning module, and an abnormal positioning sub-model.
  • the image processing method shown in FIG. 6 may include the following steps:
  • Step S601 Training the abnormal classification sub-model based on the first sample image and the abnormal category supervision label corresponding to the first sample image.
  • the number of the first sample images is multiple, and the multiple first sample images may all be positive samples, that is, all the first sample images include abnormalities, and the abnormalities corresponding to the first sample images
  • the category supervision label is used to indicate the abnormal category in each first sample image.
  • the multiple first sample images may include positive samples and negative samples, that is, a part of the first sample images in the multiple first sample images may include positive samples and negative samples.
  • the sample image includes anomalies, and there is no anomaly in a part of the first sample image.
  • the abnormal category supervision label is used to indicate that the negative sample does not include anomalies.
  • the implementation manner of training the abnormal classification sub-model based on the first sample image and the abnormal category supervision label corresponding to the first sample image may be: calling the abnormal classification sub-model to download the first sample image Sample anomaly classification processing to obtain the first anomaly category label output by the anomaly classification sub-model; determine the value of the first loss function based on the first anomaly category label and the anomaly category supervision label; follow the direction of decreasing the value of the first loss function Adjusting the parameters of the abnormality classification sub-model until the value of the first loss function reaches a minimum, then it is determined that the current abnormality classification sub-model has the highest priority.
  • the first loss function at this time may be a cross-entropy cost function.
  • Step S602 Based on the abnormal category training label corresponding to the second sample image and the second sample image, the corresponding first positioning image trains the abnormal positioning sub-model.
  • the second sample image may be the same as the first sample image or different from the first sample image.
  • the abnormal category training label corresponding to the second sample image is used to reflect the abnormal category existing in the second sample image, and the abnormal category training label may be called the abnormal classification sub-model trained in step S601 for downsampling. Exception classification processing.
  • the abnormal category training label corresponding to the second sample image may also be obtained by calling other models capable of performing abnormality recognition to perform abnormal classification processing on the second sample image.
  • step S602 may include:
  • the first positioning image and the second positioning image are used to reflect the attribute information of the prediction area associated with the abnormal category training tag.
  • the prediction area associated with the abnormal category training label refers to an abnormal area where an abnormality is predicted to occur.
  • the training feature map in step 1) may be an image obtained by the last down-sampling process in the process of down-sampling the abnormal classification of the second sample image by the anomaly classification sub-model.
  • the first category identifier may be an abnormal category number
  • the server may pre-set at least one set of correspondences between the abnormal category and the abnormal category identifier, and when the abnormal category training label is obtained, the abnormal category training label indicates The abnormal category, further, searching for the category identifier corresponding to the determined abnormality in the preset correspondence relationship, and determining the found category identifier as the first category identifier.
  • the specific implementation in step 3) may include: obtaining a first feature weight vector corresponding to the first category identifier; performing weighting processing on the first feature weight vector and the training feature image, Obtain the first positioning image.
  • the initial positioning module may include a category identification acquisition module and a feature filter.
  • the initial positioning module acquires the first category number through the category identification acquisition module, and the feature filter acquires the first category number from the training feature image. Identify the corresponding first feature weight vector.
  • the implementation manner of acquiring the first feature weight vector corresponding to the first category identifier is the same as the implementation manner of acquiring the target feature weight vector corresponding to the target category identifier in the embodiment shown in FIG. Please refer to the foregoing and will not be repeated here.
  • step 5 the obtaining the value of the target loss function corresponding to the abnormal location sub-model based on the second positioning image and the second sample image includes: based on the second positioning image and the second sample image.
  • the attribute information of the predicted abnormal area included in the positioning image and the second sample image are obtained to obtain the first optimized image and the second optimized image, and the abnormal classification sub-model is called to perform the respective calculations on the first optimized image and the first optimized image.
  • X represents the second sample image
  • m represents the second positioning image
  • ⁇ 1 represents the weight value of the first probability
  • the first probability is used to represent the first optimized image obtained by point-by-point multiplication of the predicted abnormal area included in the second positioning image and the second sample image.
  • the probability of being identified as an abnormal area by the abnormal classification sub-model Represents the second probability
  • ⁇ 2 represents the weight value of the second probability.
  • the second probability is used to indicate that the second optimized image is abnormally classified by the sub-model
  • the probability of identifying an abnormal area Represents the area occupied by the predicted abnormal area in the second positioning image
  • ⁇ 3 represents the weight value of the area.
  • the second positioning image output by the abnormal positioning sub-model may be a mask with the same size as the second sample image.
  • FIG. 7a a schematic diagram of a second sample image, a first optimized image, and a second optimized image provided by an embodiment of this application, where 71 represents the second sample image, and 72 represents the prediction that the second sample image includes The first optimized image obtained by point-by-point multiplication of the region and the second sample image, the prediction area can be reserved in the first optimized image; 73 represents the first optimized image obtained after removing the region corresponding to the prediction area from the second sample image The second optimized image, it can be seen that only the prediction area is removed from the second optimized image, and the non-prediction area is retained.
  • the area occupied by the prediction area in the second positioning image can be calculated by the following formula (2):
  • IxJ represents the total number of pixels included in the second positioning image
  • m(i, j) represents the pixel position coordinates.
  • ⁇ 4 represents the weight value of the smoothing function
  • the value of the smoothing function can be calculated by the following formula (3):
  • an embodiment of the present application provides a structural schematic diagram for training the target positioning sub-module.
  • FIG. 7b shows the determination of the first probability and the second The process of probability.
  • 701 represents the anomaly classification sub-model
  • 702 represents the initial location module
  • 703 represents the anomaly location sub-model
  • 704 represents the second sample image
  • 705 represents the target positioning image obtained after processing by the abnormal classification sub-model 701, the initial positioning module 702, and the abnormal positioning sub-model 703.
  • 705 represents the prediction area.
  • the pixels in the prediction area 7051 and the pixels in the second sample image 704 are multiplied point by point to obtain the first optimized image, and the anomaly classification sub-model is called to down-sample the first optimized image for anomaly classification
  • the first probability is obtained by processing, and when optimizing the abnormal location sub-model, the first probability is guaranteed to be as large as possible.
  • Remove the area corresponding to the predicted abnormal area from the second sample image to obtain the second optimized image call the abnormal classification sub-model to down-sample the second optimized image to obtain the second probability, and then optimize the target sub-model When, ensure that the second probability is as large as possible.
  • Step S603 Obtain the image to be detected, and call an abnormal classification sub-model in the image detection model to perform down-sampling abnormal classification processing on the image to be detected, to obtain an abnormal category prediction label and a target feature image.
  • Step S604 If the abnormal category prediction tag indicates that there is an abnormality in the image to be detected, the initial positioning module in the image detection model is called to perform preliminary abnormal location processing based on the abnormal category prediction tag and the target feature image to obtain the to-be-detected image.
  • the initial positioning image corresponding to the image.
  • step S603 it is possible to first detect whether the abnormal category prediction label indicates that there is an abnormality in the image to be detected; if the abnormal category prediction label indicates that there is an abnormality in the image to be detected, step S604 is executed; if the abnormal category is The predicted label indicates that it does not exist in the image to be detected, and step S604 may be used to directly output the abnormal category predicted label.
  • Step S605 Invoke the abnormal positioning sub-model in the image detection model to perform up-sampling abnormal positioning processing on the initial positioning image to obtain a target positioning image corresponding to the image to be detected.
  • Step S606 Output the abnormal category prediction label and the target location image.
  • the target positioning image may be a mask image
  • the prediction area may be displayed in the mask image, and other non-prediction areas are covered by the selected mask. The user can clearly and accurately determine the abnormal target area in the image to be detected through the mask image.
  • the abnormal category prediction label can be obtained by down-sampling the abnormality processing of the image to be detected by calling the abnormal classification sub-model, which saves manpower and is more efficient compared with the method of artificial analysis to obtain the abnormal category in the prior art; further, this application is implemented.
  • the initial abnormal location processing is performed, and then the initial location image obtained from the preliminary abnormal processing is up-sampled abnormal location processing to obtain the to-be-detected
  • the target positioning image of the abnormal target area in the image after two positioning processing, can ensure that the obtained target positioning image is more accurate.
  • an embodiment of the present application also provides a medical image processing device.
  • the medical processing device may include a medical image collector 801 and a medical image processing device.
  • the foregoing medical image analyzer 802 corresponds to the server in the foregoing embodiment, for example, the server 103 shown in FIG. 1a; the medical image collector 801 corresponds to the terminal in the foregoing embodiment, for example, the image shown in FIG. 1a Collection terminal 120; the medical image display 803 may be a terminal or a display device connected to the terminal, for example, the image display device 140 shown in FIG. 1a.
  • the medical image collector 801 is connected to a medical image analyzer 802, and the medical image analyzer 802 is connected to the medical image display 803.
  • the medical image processing device shown in FIG. 8a can collect and analyze medical images.
  • the interaction between the medical image collector 801, the medical image analyzer 802, and the medical image display 803 in FIG. 8a can be as shown in FIG. 8b.
  • FIG. 8b the process of collecting medical images and analyzing the medical images by the medical imaging device shown in Fig. 8a will be described in detail below.
  • the specific implementation may include the following steps:
  • Step S81 The medical image collector acquires medical images.
  • the medical image here is equivalent to the image to be detected in the embodiment of FIG. 2 and FIG. 6, and the medical image collector may be an endoscopic device, a CT device, and other medical devices that can inspect the human body.
  • Step S82 The medical image analyzer performs down-sampling disease classification processing on the medical image to obtain a disease category prediction label and a target feature image.
  • the disease category prediction label is equivalent to the abnormal category prediction label in the embodiment of FIG. 2 and FIG. 6.
  • Step S83 Perform preliminary disease location processing based on the disease category prediction label and the target feature image to obtain an initial location image corresponding to the medical image.
  • the disease category prediction tag is used to indicate whether there is a disease in the medical image, and if there is a disease, what type of disease is there. If the disease category prediction tag indicates that there is no disease in the medical image, that is, there is no target lesion area in the medical image, indicating that there is no need to further locate the lesion in the medical image, the medical image analyzer 802 can directly output the disease category to the medical image display 803 Prediction label; if the disease category prediction label indicates that there is a disease in the medical image, that is, there is a target lesion area in the medical image, the lesion location can be further performed, and the medical image analyzer 802 performs preliminary disease location processing based on the disease category prediction label and the target feature image , Get the initial positioning image corresponding to the medical image.
  • the medical image analyzer 802 performs preliminary disease location processing based on the disease category prediction label and the target feature image, and when the initial location image is obtained, the following operations are performed: obtain a label corresponding to the disease category prediction Call the initial positioning module to obtain the target feature weight vector corresponding to the target category identification, and superimpose the target feature weight vector with the target feature image to obtain the initial positioning image.
  • Step S84 The medical image analyzer performs up-sampling disease location processing on the initial location image to obtain a target location image corresponding to the medical image.
  • the medical image analyzer 802 calls the disease classification sub-model (equivalent to the aforementioned abnormal classification sub-model) when performing down-sampling disease classification processing on the medical image.
  • the medical image analyzer When performing up-sampling disease location processing on the initial location image, an abnormal location sub-model is called for execution; the disease classification sub-model includes at least one first sampling module, and the abnormal location sub-model includes at least one second sampling module , And the number of the first sampling module is greater than the number of the second sampling module.
  • the medical image analyzer When the medical image analyzer performs down-sampling disease classification processing on the medical image, the following operations are performed: acquiring a disease classification sub-model, the disease classification sub-model including at least one first sampling module; calling the at least one first sampling module A sampling module, which processes the medical image to extract the features included in the image to be detected; performs anomaly detection based on the extracted features to obtain the disease category prediction label; integrates the at least one first sampling module The characteristic image output by the last sampling module is determined to be the target characteristic image.
  • a disease location sub-model is obtained, the disease location sub-model includes at least one second sampling module, and the first sampling module The number of is greater than the number of second sampling modules; the at least one second sampling module is called to perform up-sampling disease location processing on the initial location image.
  • Step S85 The medical image display is used to display the disease category prediction label and the target location image.
  • the medical image analyzer 802 after the medical image analyzer 802 obtains the target location image, it can output the target location image and the disease category label to the medical image display 803, and the medical image display 803 displays the disease category label and the target location image in the user interface.
  • the medical image display 803 can also be connected to the medical image collector 801. After the medical image collector 801 collects the medical image, before sending the medical image to the medical image analyzer 802, the medical image can be sent to the medical image display 803.
  • the image display 803 displays medical images in the user interface.
  • the medical image display 803 may also display in the user interface whether to submit the medical image to the medical image analyzer 802, so that the medical image analyzer 802 performs disease identification and location processing on the medical image. . If the medical image display 803 detects the confirmation submission operation for the prompt information, the medical image display 803 sends the medical image to the medical image analyzer 802; if the medical image display 803 detects the cancel submission operation for the prompt information, then the medical The image display 803 may not send the medical image to the medical image analyzer 802.
  • the medical image collector 801 collects medical images, it sends the medical images to the medical image display 803, and the medical image display 803 displays the medical images on the user interface and whether to submit the medical images to the medical image analyzer 802. Prompt information.
  • FIG. 8c a schematic diagram of a user interface of a medical display provided by an embodiment of this application.
  • 11 represents the medical image collected by the medical image collector 801
  • 12 represents prompt information, which may include confirmation
  • the submit button 120 and the cancel submit button 121 when the user clicks the confirm submit button 120, the medical image display 803 submits the medical image to the medical image analyzer 802 for analysis and processing; when the user clicks the cancel submit button 121, the medical image display 803
  • the medical image is not submitted to the medical image analyzer 802.
  • the disease category prediction label can be obtained by down-sampling the disease classification processing of the medical image, which saves manpower and is more efficient compared with the method of artificial analysis to obtain the disease category in the prior art; further, the present invention
  • preliminary disease location processing is performed based on the disease category prediction label and the target feature image obtained in the down-sampling process; then, the initial location image obtained from the preliminary abnormality processing is subjected to up-sampling disease location processing to obtain medical treatment
  • the target positioning image of the medical image in the image, after two positioning processing, can ensure that the obtained target positioning image is more accurate.
  • an application scenario diagram of medical image processing provided by this embodiment of the present application includes a medical image collector such as an endoscope 901, a medical image display 902, and a medical image analyzer 903.
  • a medical image collector such as an endoscope 901, a medical image display 902, and a medical image analyzer 903.
  • the doctor uses the endoscope 901 to examine the human digestive tract and obtains medical video data; the endoscope 901 can send the medical video data to the medical display 902, and the medical image display 902 divides the medical video data through a script to obtain multiple Frame medical image.
  • the medical image display 902 sequentially sends multiple frames of images to the medical image analyzer 903.
  • 9b a schematic diagram of medical image processing by a medical image analyzer 903 according to an embodiment of this application.
  • the medical image analyzer 903 calls an image detection model to perform disease category identification and lesion area location processing on the medical image to obtain the disease category
  • the target location image of the predicted label and the lesion area is shown as 91 in Fig. 9b.
  • the target positioning image may include a mask positioning image 911 and a non-mask positioning image 912.
  • the medical image analyzer 903 sends the disease category prediction label and the target location image of the lesion area to the medical image display 902, which is displayed on the user interface by the medical image display 902.
  • the doctor can diagnose the disease based on the content displayed on the medical image display 902.
  • an embodiment of the present application also provides an image processing device.
  • the image processing device can run the following units:
  • the acquiring unit 1001 is used to acquire the image to be detected
  • the processing unit 1002 is configured to perform down-sampling abnormal classification processing on the image to be detected to obtain an abnormal category prediction label and a target feature image; perform preliminary abnormal location processing based on the abnormal category prediction label and the target feature image to obtain the to-be-detected image An initial positioning image corresponding to the image; performing up-sampling abnormal positioning processing on the initial positioning image to obtain a target positioning image corresponding to the image to be detected;
  • the output unit 1003 is configured to output the abnormal category prediction label and the target location image. Wherein, the initial positioning image and the target positioning image are used to reflect the attribute information of the target area associated with the abnormal category prediction tag.
  • the processing unit 1002 is configured to obtain the target category identifier corresponding to the abnormal category prediction label; call the initial positioning module to obtain the target feature weight vector corresponding to the target category identifier, and to The target feature weight vector and the target feature map are superimposed to obtain the initial positioning image.
  • the processing unit 1002 is configured to obtain an anomaly classification sub-model, the anomaly classification sub-model includes at least one first sampling module; calling the at least one first sampling module to perform an analysis on the image to be detected Perform processing to extract the features included in the image to be detected; perform anomaly detection based on the extracted features to obtain the abnormal category prediction label; and extract the feature image output by the last sampling module in the at least one first sampling module, Determined as the target feature image.
  • the processing unit 1002 is configured to obtain an abnormal location sub-model, the abnormal location sub-model includes at least one second sampling module, and the number of the first sampling module is greater than the number of the second sampling module; The at least one second sampling module performs up-sampling abnormal positioning processing on the initial positioning image.
  • the processing unit 1002 is further configured to obtain a second sample image; call the abnormal classification sub-model, and perform down-sampling abnormal classification processing on the second sample image to obtain an abnormal category training label; call The initial positioning module performs preliminary abnormal positioning processing based on the abnormal category training tag to obtain a first positioning image; trains to obtain the abnormal positioning sub-model based on the abnormal category training tag and the first positioning image.
  • the processing unit 1002 is further configured to perform down-sampling abnormal classification processing on the second sample image to obtain a training feature image; obtain the first category identification corresponding to the abnormal category training label; The initial positioning module performs preliminary abnormal positioning processing based on the first category identifier and the training feature image to obtain the first positioning image.
  • the units in the image processing device shown in FIG. 10 can be combined separately or all into one or several other units to form, or some unit(s) of them can also be split. It is composed of multiple units with smaller functions, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application.
  • the above-mentioned units are divided based on logical functions. In practical applications, the function of one unit may also be realized by multiple units, or the functions of multiple units may be realized by one unit. In other embodiments of the present application, the image-based processing device may also include other units. In actual applications, these functions may also be implemented with the assistance of other units, and may be implemented by multiple units in cooperation.
  • a general-purpose computing device such as a computer including a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM) and other processing elements and storage elements.
  • CPU central processing unit
  • RAM random access storage medium
  • ROM read-only storage medium
  • a computer program (including program code) that can execute each step involved in the corresponding method shown in FIG. 2 and FIG. 6 to construct the image processing apparatus shown in FIG. Image processing method.
  • the computer program may be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium, and run in it.
  • an embodiment of the present application also provides a server.
  • FIG. 11 is a schematic structural diagram of a server provided in an embodiment of the present application.
  • the server shown in FIG. 11 may include a processor 1101 and a computer storage medium 1102.
  • the computer storage medium 1102 may be stored in the memory of the terminal.
  • the computer storage medium 1102 is used to store a computer program.
  • the computer program includes program instructions.
  • the processor 1101 is used to execute the program instructions stored in the computer storage medium 1102. .
  • the processor 1101 or CPU Central Processing Unit, Central Processing Unit
  • the processor 1101 or CPU is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to realize the corresponding method flow Or a corresponding function; in an embodiment, the processor 1101 may be used to execute the image processing method described in the foregoing embodiment.
  • the embodiment of the present application also provides a computer storage medium (Memory).
  • the computer storage medium is a memory device in an electronic device for storing programs and data. It can be understood that the computer storage medium herein may include a built-in storage medium in an electronic device, and of course, may also include an extended storage medium supported by the electronic device.
  • the computer storage medium provides storage space, and stores one or more instructions suitable for being loaded and executed by the processor 1101 in the storage space. These instructions may be one or more computer programs (including program codes).
  • the computer storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory; optionally, it can also be at least one located far away from the aforementioned processor.
  • Computer storage media can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk memory; optionally, it can also be at least one located far away from the aforementioned processor.
  • the program can be stored in a computer readable storage medium, and the program can be stored in a computer readable storage medium. During execution, it may include the procedures of the above-mentioned method embodiments.
  • the storage medium may be a magnetic disk, an optical disc, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Medical Informatics (AREA)
  • Quality & Reliability (AREA)
  • Radiology & Medical Imaging (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Multimedia (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Databases & Information Systems (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例公开了一种图像处理方法、装置、服务器、医疗图像处理设备及存储介质，其中方法包括：获取待检测图像，并对待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图；基于异常类别预测标签以及目标特征图像进行初步异常定位处理，得到待检测图像对应的初始定位图像；对初始定位图像进行上采样异常定位处理，得到待检测图像对应的目标定位图像；输出异常类别预测标签和目标定位图像。

Description

图像处理方法、装置、服务器、医疗图像处理设备及存储介质
本申请要求于2019年11月12日提交中国专利局、申请号为201911103557.7、申请名称为“图像处理方法、装置、服务器及存储介质”的中国专利申请的优先权。
技术领域
本申请涉及人工智能技术的图像处理领域，具体涉及图像处理方法、装置、服务器、医疗图像处理设备及存储介质。
发明背景
目前，通过对包括异常区域的图像进行分析以确定异常区域所属的异常类别以及发生异常的具体位置是目前常用的异常分类方法。例如，在医学领域中，通过采集内窥镜设备对病人进行内镜检查得到的图像，医生通过对图像进行分析处理可以确定病人所患疾病以及发病位置；在车辆检测领域中，维修人员或其他技术人员通过对包括车辆故障位置图像进行分析处理，可以确定车辆故障所述的故障类型，以及发生该或引起该故障的具体位置等。
在上述异常分类方法中，通常是专门的技术人员对采集到的图像进行分析处理，从而确定异常类型以及发生异常的具体位置。如此一来，在异常分类过程中需要耗费大量的人力，并且检测效率不高。目前，随着计算机视觉技术的兴起，将计算机视觉技术应用在图像检测中成为图像检测的新趋势，因此，当今如何基于计算机视觉技术进行高效的图像检测成为研究的热点问题。
发明内容
本申请实施例提供了一种图像处理方法、装置、服务器、医疗图像处理设备及存储介质，可以较高效地对待检测图像进行异常检测。
一方面，本申请实施例提供了一种图像处理方法，由服务器执行，包括：
获取待检测图像，并对所述待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像；
基于所述异常类别预测标签以及所述目标特征图像进行初步异常定位处理，得 到所述待检测图像对应的初始定位图像；
对所述初始定位图像进行上采样异常定位处理，得到所述待检测图像对应的目标定位图像；
输出所述异常类别预测标签和所述目标定位图像；
其中，所述初始定位图像和所述目标定位图像用于反映所述异常类别预测标签关联的目标区域的属性信息。
另一方面，本申请实施例提供了一种图像处理装置，包括：
获取单元，用于获取待检测图像；
处理单元，用于对待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像；基于所述异常类别预测标签和所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像；对所述初始定位图像进行上采样异常定位处理，得到所述待检测图像的目标定位图像；
输出单元，用于输出所述异常类别预测标签和所述目标定位图像；
其中，所述初始定位图像和所述目标定位图像用于反映所述异常类别预测标签关联的目标区域的属性信息。
另一方面，本申请实施例提供了一种服务器，所述服务器包括：
处理器，适于实现一条或多条指令；以及，
计算机存储介质，所述计算机存储介质存储有一条或多条指令，所述一条或多条指令适于由所述处理器加载并执行上述图像处理方法。
另一方面，本申请实施例提供了一种医疗图像处理设备，所述医疗图像采集器、医疗图像分析器以及医疗图像显示器，所述医疗图像采集器与所述医疗图像分析器相连接，所述医疗图像显示器与所述医疗图像分析器相连；
所述医疗图像采集器，用于获取医疗图像；
所述医疗图像分析器，用于对所述医疗图像进行下采样疾病分类处理，得到疾病类别预测标签和目标特征图像；基于所述疾病类别预测标签以及目标特征图像进行初步疾病定位处理，得到所述医疗图像对应的初始定位图像；所述医疗图像分析 器，还用于对所述初始定位图像进行上采样疾病定位处理，得到所述医疗图像对应的目标定位图像；将所述疾病类别预测标签和所述目标定位图像输出至所述医疗图像显示器；
所述医疗图像显示器，用于显示所述疾病类别预测标签和所述目标定位图像；
其中，所述初始定位图像和所述目标定位图像用于反映所述疾病类别预测标签关联的目标区域的属性信息。
另一方面，本申请实施例提供了一种计算机存储介质，所述计算机存储介质存储有一条或多条指令，所述一条或多条指令适于由处理器加载并执行上述图像处理方法。
另一方面，本申请实施例还提供了一种计算机程序产品或计算机程序，该计算机程序产品或计算机程序包括计算机指令，该计算机指令存储在计算机可读存储介质中。计算机设备的处理器从计算机可读存储介质读取该计算机指令，处理器执行该计算机指令，使得该计算机设备执行上述图像处理方法。
附图简要说明
为了更清楚地说明本申请实施例技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1a是本申请实施例提供的一种图像处理系统的结构示意图；
图1b是本申请实施例提供的一种异常定位模型的结构示意图；
图2是本申请实施例提供的一种图像处理方法的流程示意图；
图3是本申请实施例提供的一种确定目标特征权重向量的示意图；
图4a是本申请实施例提供的一种初始定位图像的示意图；
图4b是本申请实施例提供的一种目标定位图像的示意图；
图5是本申请实施例提供的另一种图像检测模型的结构示意图；
图6是本申请实施例提供的另一种图像处理方法的流程示意图；
图7a是本申请实施例提供的一种第二样本图像以及优化图像的示意图；
图7b是本申请实施例提供的一种确定第一概率和第二概率的示意图；
图8a是本申请实施例提供的一种医疗图像处理设备的结构示意图；
图8b是本申请实施例提供的一种医疗图像处理设备进行图像处理示意图；
图8c是本申请实施例提供的一种医疗图像显示器的用户界面的示意图；
图9a是本申请实施例提供的一种医疗图像处理的应用场景图；
图9b是本申请实施例提供的一种对医疗图像处理的示意图；
图10是本申请实施例提供的一种图像处理装置的结构示意图；
图11是本申请实施例提供的一种服务器的结构示意图。
实施方式
随着人工智能技术的研究和进步，人工智能技术在多个领域展开研究和应用，例如常见的智能家居、智能穿戴设备、智能医疗以及智能音箱等，相信随着技术的发展，人工智能技术将在更多的领域得到应用，并发挥越来越重要的价值。本申请实施例提供的方案涉及人工智能技术中的计算机视觉技术，所述计算机视觉(Computer Vision，CV)技术能够利用摄影机和智能终端代替人眼对目标对象进行识别、跟踪和测量并处理输出更适合人眼观察或者传送给特定仪器检测的图像。下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述。
在对通过图像进行异常检测的研究中发现，可以预先训练用于检测异常类别和异常定位的图像检测模型，当需要对待检测图像进行检测时，调用预先训练完成的图像检测模型对待检测图像进行分析处理，便可得到待检测图像包括的异常类别以及发生异常的异常区域的属性信息，所述属性信息可以包括异常区域的具体位置和大小。其中，在不同应用领域中，所述异常区域的含义不同。例如，在医学领域中，所谓异常区域可以指发生病变的区域；在车辆领域中，所述异常区域可以指发生故障的车辆组件；在其他领域中，所述异常区域可以指特定物体所在区域，比如待检测图像中包括云朵、太阳以及飞机，可以将云朵、太阳或者飞机所在位置称为待检测图像的异常区域。
传统技术中，对图像检测模型进行训练，是基于常规的有监督学习训练图像检测模型。所谓基于常规的有监督学习训练图像检测模型是指训练样本对应的标签与图像检测模型所具有的功能具有相同粒度。但是，这种模型需要人工为训练样本标注异常类别标签和异常定位标签，工作量大且繁琐。
此外，在某些领域，对异常定位的标注是很难实现的，例如在医学领域中，对病变位置和大小的标注需要更多的人力物力，实现难度系数较高，一旦标注出错就会严重影响图像检测模型异常定位的准确性。
鉴于此，本申请实施例中，采用基于弱监督学习的训练方式进行图像检测模型的训练。所谓弱监督学习是指训练样本对应的标签的粒度低于与图像检测模型所具有的功能的粒度。比如图像检测模型的功能是对内窥镜图像中疾病进行分类识别和对发生疾病的病变区域定位，此时训练样本只有对应的疾病类别标签即可。与上述传统方法相比，该种方法省去了标注定位标签的步骤，可以节省人力物力，同时也避免了由于定位标签出错对图像检测模型准确性的影响。
图1a是本申请实施例提供的一种图像处理系统的结构示意图。如图1a所示，图像处理系统100包括用户101、图像采集终端120、服务器103以及图像显示设备104。其中，服务器103包括处理器和存储器，本申请中的方法实施例由处理器执行存储在存储器中的指令来执行。具体地，服务器103包括图像处理装置1031。
图像采集终端120是指具备图像采集功能的任何一种智能终端。当采集医学图像时，图像采集终端120具体为带有摄影机的医学采集设备，例如，内窥镜设备，对用户101进行内窥镜检测，内窥镜设备上配置的视频采集模块采集内窥镜视频数据，并将采集到的一帧帧图像发送给服务器103。
服务器103对目标对象进行识别、跟踪和检测，具体地，其中的图像处理装置1031可以对待检测图像进行处理，从中得到异常类别预测标签和目标定位图像，发送给图像显示设备104进行显示。例如，显示给医生检测结果，获知用户101的检测部位是否异常，以及异常时发生病变的那部分图像。
其中，服务器110可以是一台服务器，或者由若干台服务器组成的服务器集群，或者是一个云计算服务中心。
参考图1b，为本申请实施例提供的一种图像检测模型的结构示意图。图1b所 示的结构示意图可包括异常分类子模型101、初始定位模块102以及异常定位子模型103，异常分类子模型101用于进行下采样异常分类处理，初始定位模块102用于基于下采样异常处理结果进行初步异常定位，异常定位子模型103用于基于初步异常定位的结果进行上采样异常定位处理。
在一个实施例中，异常分类子模型101可以是已经训练完成的具有异常分类能力的模型，可以直接用于进行下采样异常处理；或者，异常分类子模型101也可以是未训练的模型。由于初始定位模块102和异常定位子模型103在进行异常定位时，都需要直接或间接依赖异常分类子模型101的处理结果，因此，为了保证异常定位的准确性，首先要保证异常分类子模型101进行下采样异常分类处理的结果较为准确。如果异常分类子模型101是未训练的模型，在对图像检测模型进行训练时，首先要对异常分类子模型101进行训练。
在一个实施例中，异常分类子模型101可以为具有图像识别功能的任意一种图像识别模型。可选的，卷积神经网络模型由于其较强的特征提取性能成为目前常用的图像识别模型，本申请实施例中，异常分类子模型101可以为卷积神经网络模型，例如res-net模型、dense-net模型、VGG以及GoogleNet模型中的任意一个。
在一个实施例中，卷积神经网络模型主要包括负责特征提取的全卷积层、用于缩减输入数据规模的池化层、以及根据池化层的输出计算分类的分类值的全连接层，以及进一步进行分类处理并输出分类结果的softmax层。所述池化层可以包括全局平均池化层、最大池化层、重叠池化层等，在不同卷积神经网络模型中池化层不同。其中，每个卷积层负责提取不同的特征，得到特征图像，前一个卷积层得到的特征图像作为后一个卷积层的输入。每个卷积层负责提取的特征可以是根据特定函数设定的，或者是人为设定的。
例如，对于图形类的图像识别时，可以设定第一卷积层负责提取图形的整体形状特征；第二卷积层负责提取图形的线条特征；第三卷积层负责提取图形的非连续性特征等。再如，对于包含人脸的图像识别时，可以设定第一卷积层负责提取人脸的轮廓特征；第二卷积层负责提取人脸的五官特征。每个卷积层包括多个特征提取通道，每个特征提取通道中包括多个相同尺寸的用于进行卷积计算的滤波器。
如图1b中所示，异常分类子模型101具体包括：全卷积层(也叫下采样层)和 softmax层。其中，全卷积层中包括多个卷积层。在本申请实施例中，异常分类子模型101还包括全局平均池化层和全连接层(图1b中未示出)，该全局平均池化层位于全卷积层和全连接层之间，全连接层与softmax层连接。
可选的，异常分类子模型101是基于第一样本图像和所述第一样本图像对应的异常类别监督标签训练得到的，对于具体的训练过程将在后面详细描述。
在一个实施例中，对异常分类子模型101训练完成后，基于训练完成的异常分类子模型101以及初始定位模块102对目标定位子模型103进行训练。
具体的过程可以是：获取第二样本图像，该第二样本图像可以与第一样本图像相同，也可以不同；调用训练完成的异常分类子模型101对第二样本图像进行异常分类处理，得到异常类别训练标签；并提取在对第二样本图像进行异常分类处理过程中产生的训练特征图像，该训练特征图像可以是异常分类子模型101的全卷积层的最后一个卷积层进行特征提取得到的。
紧接着，初始定位模块102块基于训练特征图像和异常类别训练标签进行异常定位处理，得到对第二样本图像中异常区域进行异常定位的第一定位图像。其中，初始定位模块102可包括特征过滤器和类别标识获取模块两部分，特征过滤器用于对训练特征图像进行处理，类别标识获取模块用于获取与异常类别训练标签对应的类别标识，该类别标识可以是异常类别训练标签所指示的异常类别对应的标识号，比如标识号0对应无异常，标识号1对应异常类型1，标识号2对应异常类型2。
进一步的，调用异常定位子模型103对第一定位图像进行异常定位处理，得到第二定位图像；最后，根据第二定位图像和第二样本图像确定用于训练异常定位子模型103的目标损失函数的值，按照减小该目标损失函数的值的方向优化异常定位子模型103，最终得到训练完成的异常定位子模型。
在一个实施例中，对异常定位子模型103训练完成，也就意味着图像检测模型训练完成。当获取到待检测图像时，可以调用训练完成后的图像检测模型中各个子模型或者模块对待检测图像进行处理，得到待检测图像中包括的异常类别以及发生异常的异常位置和异常区域大小。
基于上述的图像检测模型，本申请实施例提供了一种图像处理方法。参考图2， 为本申请实施例提供的一种图像处理方法的流程示意图。图2所示的图像处理方法可以由服务器执行，具体可由服务器的处理器执行，例如，图1a中所示的服务器103。图2所示的图像处理方法可以包括以下步骤：
步骤S201、获取待检测图像，并对待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像。
在一个实施例中，所述待检测图像可以是终端发送的，比如医生通过内窥镜设备对病人进行内窥镜检测，内窥镜设备上配置的视频采集模块采集内窥镜视频数据，内窥镜设备或者与内窥镜相连的终端可以将视频数据进行图像分帧处理，得到多帧图像；进一步的，将得到的多帧图像作为待检测图像发送给服务器；在其他实施例中，所述待检测图像也可以是服务器接收终端发送的视频数据，对视频数据进行图像分帧处理得到的。
其中，所述异常类别预测标签用于反映待检测图像中是否存在异常，以及如果存在异常，异常所属的异常类型。举例来说，如果待检测图像为医疗图像，异常类别预测标签可以指示待检测图像中是否存在疾病，如果存在疾病，该疾病属于何种类型疾病，比如肿瘤、胃癌、肝硬化等。异常类别预测标签可以包括用于表示某种异常的目标类别标识，或者异常类别预测标签还可以包括存在该种异常类别的概率。例如，预先设置无异常的目标类别标识为0，异常类别1的类别标识为1，异常类别2的类别标识为2。基于上述描述，所述异常类别预测标签可以为(1，60％)、(2，40％)，其中，数字1、2标识类别标识，后面百分数表示存在相应异常类别的概率。
在一个实施例中，上述异常类别预测标签的数量为一个。如果对待检测图像进行异常分类，得到多个异常分类预测标签，则从中选择置信度最高的作为待检测图像对应的异常类别预测标签。
在一个实施例中，所述对待检测图像进行下采样异常处理的目的是为了确定待检测图像中是否存在异常，如果存在异常，具体是何种类别的异常。在一个实施例中，对待检测图像进行下采样异常处理的方式可以包括：通过至少一个卷积模块对所述待检测图像进行卷积处理，提取出待检测图像包括的特征；基于提取到的特征进行异常检测，得到异常类别预测标签。可选的，上述至少一个卷积模块可以是直接配置于服务器中的。
或者，服务器中可运行异常分类子模型，当采用异常分类子模型对待检测图像进行下采样异常分类处理时，具体包括：获取异常分类子模型，所述异常分类子模型包括至少一个第一采样模块；调用至少一个第一采样模块，对所述待检测图像进行卷积处理，提取出所述待检测图像包括的特征；基于提取到的特征进行异常检测，得到所述异常类别预测标签；将所述至少一个第一采样模块中最后一个采样模块输出的特征图像，确定为所述目标特征图像。
如上述图1b所示，所述异常分类子模型中可包括卷积层，上述至少一个采用模块具体为卷积模块，对应了异常分类子模型的卷积层。在下面的描述中，均以卷积模块为异常分类子模型的卷积层为例进行描述。
由前述可知，所述异常分类子模型主要用于对待检测图像进行异常分类，得到待检测图像的异常类别预测标签。可以基于第一样本图像和第一样本图像对应的异常类别监督标签，训练得到异常分类子模型。所述异常类别监督标签反映了第一样本图像中是否包括异常以及如果包括异常，所述异常所属的异常类别。所述异常类别监督标签可以是人工对第一样本图像进行标注得到的；或者，所述异常类别监督标签也可以是通过人工标注和标注模型共同标注得到的。
所述异常分类子模型的每个卷积层包括多个特征提取通道，每个特征提取通道包括多个尺寸相同的用于卷积计算的滤波器。基于此，所述对待检测图像进行下采样异常处理的方式可以包括：通过至少一个卷积模块对所述待检测图像进行卷积处理，包括：通过至少一个卷积模块中的多个滤波器对待检测图像进行卷积处理。
具体实现中，对待检测图像进行卷积处理是指每个卷积模块对待检测图像进行卷积处理得到对应的特征图像；将该特征图作为下一个卷积模块的输入，以使得下一个卷积模块对该特征图进行卷积处理，以此类推，直至最后一个卷积模块进行卷积处理后，输出最后一层的特征图像，该最后一层的特征图像中包含了待检测图像的空间结构信息。
在步骤S201中，所述目标特征图像是指在对待检测图像进行下采样异常分类处理过程中产生的特征图像，为了后续基于目标特征图像和异常类别预测标签进行初步异常定位处理时，得到准确性较高的初始定位图像，本申请实施例中，将最后一个卷积模块输出的特征图像确定为目标特征图像。
步骤S202、基于异常类别预测标签以及目标特征图像进行初步异常定位处理，得到待检测图像对应的初始定位图像。
其中，初始定位图像可以用于粗略反映异常类别预测标签关联的目标区域的属性信息，所述属性信息包括目标区域的位置和大小。其中，如果异常类别预测标签指示待检测图像中不存在异常，则所述异常类别预测标签关联的目标区域不存在；如果异常类别预测标签指示待检测图像中存在某种类型的异常，则所述异常类别预测标签关联的目标区域是指发生该种类型异常的区域。可选的，目标区域的属性信息可以是以标注框的形式显示的，标注框的位置表示发生异常的区域所在位置，标注框的大小表示发生异常的区域的大小。
在一个实施例中，所述基于异常类别预测标签以及目标特征图像进行初步异常定位处理，得到待检测图像对应的初始定位图像的实施方式可以是：基于异常类别预测标签对应的目标类别标识以及目标特征图像进行初步异常定位处理，得到所述初始定位图像。其中，所述异常类别预测标签对应的目标类别标识是用来唯一标识异常类别预测标签所指示的异常的，该目标类别标识可以是异常类别号，异常类别号可以是预先设置的，比如在医学领域中，可以设置为异常类别号0表示无病变，异常类别号1表示胃炎，异常类别号2表示肠炎，异常类别号3表示胃癌等等。
具体可选地，步骤S302中所述基于所述异常类别预测标签以及所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像，包括：获取与所述异常类别预测标签对应的目标类别标识；调用初始定位模块，获取与所述目标类别标识对应的目标特征权重向量，并将所述目标特征权重向量与所述目标特征图像进行叠加处理，得到所述初始定位图像。
其中，所述初始定位模块可以是配置于服务器中的模块，也可以是独立运行的模块。可选的，初始定位模块可以包括特征过滤器和类别标识获取模块，所述初始定位模块可以通过类别标识获取模块获取目标类别标识，特征过滤器用于获取与目标类别标识对应的目标特征权重向量。
在一个实施例中，获取与所述异常类别预测标签对应的目标类别标识的实施方式可以为：服务器可以预先设置多个异常类别与类别号之间的对应关系，当通过对待检测图像进行下采样异常分类处理得到异常类别预测标签时，确定所述异常类别 预测标签所指示的异常，进一步的，类别标识获取模块在预先设置的对应关系中查找与确定出的异常对应的目标类别标识。
在其他实施例中，所述获取与所述异常类别预测标签对应的目标类别标识的实施方式还可以为：类别标识获取模块接收终端发送的目标类别标识。应当理解的，当在步骤S201中确定出异常类别预测标签之后，可以将异常类别预测标签发送给终端，由终端将异常类别预测标签显示在终端的用户界面中以供用户查看。如果用户想要进一步对待检测图像进行异常定位，可以在终端的用户界面中输入定位处理的触发操作，例如终端的用户界面中可包括异常定位的触发选项，当用户点击该选项时表明用户输入了定位处理的触发操作。用户在输入触发操作的同时可以在用户界面中选择或输入与异常类别预测标签对应的目标类别标识；终端在检测到触发操作时，将用户输入的目标类别标识发送给类别标识获取模块。
在一个实施例中，如果异常类别预测标签的数量为至少两个，则用户在用户界面中输入定位处理的触发操作时，还应该指出对哪个异常类别预测标签所表示的异常进行定位处理。
由前述可知，异常分类子模型还可以包括全连接层和池化层，池化层用于对最后一个卷积层输出的目标特征图进行缩减处理，例如最后一个卷积层输出的目标特征图像表示为[N，7，7，2048]，N表示特征提取通道数，2048表示每个特征提取通道中滤波器的个数，7x7表示滤波器的尺寸，将该目标特征图像经过全局平均池化后得到的特征图像为[N，1，1，2048]。全连接层用于根据池化后的目标特征图像计算分类值。
在一个实施例中，为了得到与目标类别标识对应的目标特征权重向量，一般有两种方法：一种是非训练的方法，另一种是训练的方法。参考图3，为本申请实施例提供的一种通过非训练法确定目标特征权重向量和初始定位图像的示意图。
在图3中，将输入的待检测图像经过全卷积层，得到目标特征图像；然后将目标特征图像进行全局平均池化，最后输入到全连接层得到分类结果。目标特征图像经过全局平均池化之后，对于每一个异常类别x来说，池化后每个通道的目标特征图像都有一个对应的权重参数，可以记为
Figure PCTCN2020115768-appb-000001
其中x表示一个异常类别，k表示池化后每个通道的目标特征图像，k的取值可以为1到N之间，N表示通道数。进一 步的，从上述
Figure PCTCN2020115768-appb-000002
中选择与目标类别标识所指示的异常关联的权重参数，作为目标特征权重向量。例如，目标类别标识所指示的异常类别为c，从
Figure PCTCN2020115768-appb-000003
中选择c这个异常类别对应的权重参数作为目标特征权重向量，可以记为
Figure PCTCN2020115768-appb-000004
进一步的，将异常类别c对应的目标特征权重向量与池化之前的目标特征图像进行加权处理，便可得到热力图，该热力图即为初始定位图像。
在一个实施例中，在将目标特征权重向量和目标特征图像进行加权处理后，还加了一个ReLU，加入ReLU可以过滤掉其他不属于异常类别c的像素点，可以增强对异常类别c有影响的像素点。
在图3中确定目标特征权重向量时，全连接层采用的是全局平均池化的全连接层，而非传统的全连接层，这是由于传统全连接层将全卷积层的多层的神经元直接拉伸为一个向量，破坏了像素之间的空间结构信息，且参数量巨大。采用全局平均池化的方法能够保留目标特征图像所蕴含的空间信息，有利于对目标异常区域进行定位。
对于通过训练法获取目标特征权重向量，主要借鉴embedding的思想，embedding的本质也是学习一种映射关系，即从语义空间(类别号)到向量空间(特征图各通道的权重向量)的映射。举例来说，假设总的异常类别的数量为m，embedding参数为
Figure PCTCN2020115768-appb-000005
尺寸为[m，N]。假设目标类别标识为向量selector(例如目标类别标识0：[0])，则embedding操作后得到的向量embedding_vec尺寸为[1，N]，也就是异常类别0对应映射到N长向量空间中的向量，输入不同的类别标识，经过embedding操作后可以得到N长向量空间中不同的向量。该N长向量和目标特征图F_n(x,y)相乘后得到激活向量，即activation_vec(x,y)＝sigmoid(F_n(x,y)*embedding_vec)，该向量可以理解为对应异常在特征图上的权重向量。最后activation_vec*F_n(x,y)加权即得到初步定位图像。
步骤S203、对初始定位图像进行上采样异常定位处理，得到待检测图像对应的目标定位图像。
相比于步骤S202中的初始定位图像，目标定位图像可以用于较为精确的反映出异常类别预测标签关联的目标区域的属性信息。参考图4a为本申请实施例提供的一种初始定位图像的示意图，图4b为本申请实施例提供的一种目标定位图像的示意 图，在图4a中401表示目标区域，图4b中402表示目标区域，由图可见，在图4a中除了隐约看到目标区域外，还可以看到其他区域403；但是，在图4b中能够清晰的看到目标区域的位置和大小，并且看不到其他区域。
在一个实施例中，所述对初始定位图像进行上采样异常定位处理，得到目标定位图像的实施方式可以为：获取异常定位子模型，所述异常定位子模型包括至少一个第二采样模块，且所述异常分类子模型中包括的第一采样模块的数量大于第二采样模块的数量；调用至少一个第二采样模块，对初始定位图像进行上采样异常定位处理，得到目标定位图像。
在一个实施例中，获取异常定位子模型的具体方式包括：获取第二样本图像；调用所述异常分类子模型，对第二样本图像进行下采样异常分类处理，得到异常类别训练标签；调用所述初始定位模块，基于所述异常类别训练标签进行初步异常定位处理，得到第一定位图像；基于所述异常类别训练标签以及第一定位图像，训练得到异常定位子模型。其中，训练的具体过程，将在后面的实施例中进行阐述。
在一个实施例中，上述异常分类子模型、初始定位模块以及异常定位子模型可以组成一个图像检测模型，所述图像检测模型可以用于对待检测图像进行检测，输出待检测图像对应的异常类别预测标签以及待检测图像对应的目标定位图像。在其他实施例中，异常分类子模型、初始定位模块以及异常定位子模型也可以是三个独立的模型，服务器可以调用异常分类子模型执行步骤S201，调用初始定位模型执行步骤202以及调用异常定位子模型执行步骤S203。在本申请实施例的下面描述中，以异常分类子模型、初始定位模块以及异常定位子模型组成一个图像检测模型为例，该图像检测模型的示意图可如图1b所示。
步骤S204、输出异常类别预测标签和目标定位图像。
在一个实施例中，服务器在得到待检测图像对应的目标定位图像之后，可以将异常类别预测标签以及目标定位图像一起输出给终端，由终端显示在用户界面上以供用户查看。
在一个实施例中，假设在步骤S201中，通过调用图1b所示图像检测模型中的异常分类子模型对待检测图像进行下采样异常分类处理得到异常类别预测标签以及目标特征图像，所述下采样异常分类处理包括异常分类子模型的多个卷积模块对待 检测图像进行卷积处理，所述目标特征图像可以是最后一个卷积模块进行卷积处理得到的。在步骤S202中基于目标特征图和异常类别预测标签得到的初始定位图像分辨率较低，需要通过步骤S203进一步的对初始定位图像进行上采样精修处理，得到分辨率较高的目标定位图像。
在一个实施例中，为了得到较高分辨率较为清晰的目标定位图像，异常分类子模型中用于下采样处理的卷积模块的数量应该大于异常定位子模型中用于上采样处理的上采样模块的数量。举例来说，参考图5，为本申请实施例提供的另一种图像检测模型的结构示意图，501表示异常分类子模型、502表示初始定位模块、503表示异常定位子模型。假设异常分类子模型501为卷积神经网络模型中的resent-50模型，resent-50模型中包括5个卷积模块和softmax层，待检测图像的尺寸为224*224，经过5个卷积模块进行5次下采样处理之后，得到目标特征图的尺寸为7*7。初始定位模块对目标特征图像进行初始定位处理得到初始定位图像后尺寸不变仍为7*7；进一步的，异常定位子模型503对初始定位图像进行3次上采样处理，得到的目标定位图像的尺寸为56*56，边长仅为待检测图像的1/4，然后经过插值的方法将目标定位图像的尺寸恢复到224*224。
不直接在目标定位子模块采用5个上采样模块是因为图5所示的图像检测模型中在异常分类子模型501和异常定位子模型503之间有捷径的存在，采用该种结构可以融合待检测图像以便于进行特征对齐。另外，越上层的特征图像包括的都是较为底层的语义，异常定位子模型503进行定位时需要更多的具有判断力的高层语义信息，过多底层语义会使定位不准确。
在上述的图像检测过程中，可以通过对待检测图像进行下采样异常处理得到异常类别预测标签，与传统技术中人为分析得到异常的方法相比，节省人力并且更加高效；进一步的，本申请实施例中，基于异常类别预测标签和下采样异常处理过程中获取到的目标特征图像进行初步异常定位处理，紧接着，再对初步异常处理得到的初始定位图像进行上采样异常定位处理，得到待检测图像对应的目标定位图像，经过两次定位处理，可以保证得到的目标定位图像更加准确。
基于上述的图像处理方法，本申请实施例还提供了另一种图像处理方法，参考 图6，为本申请实施例提供的另一种图像处理方法的流程图。图6所示的图像处理方法可由服务器以及服务器调用图像检测模型执行，例如，图1a中所示的服务器103，所述图像检测模型可包括异常分类子模型、初始定位模块以及异常定位子模型。图6所示的图像处理方法可包括如下步骤：
步骤S601、基于第一样本图像和第一样本图像对应的异常类别监督标签，对异常分类子模型进行训练。
其中，第一样本图像的数量为多个，多个第一样本图像可以均为正样本，也即所有的第一样本图像中均包括异常，此时第一样本图像对应的异常类别监督标签用于表示各个第一样本图像中的异常类别。在其他实施例中，为了提高异常分类子模型能够准确地进行异常分类，多个第一样本图像中可以包括正样本和包括负样本，也即多个第一样本图像中有一部分第一样本图像中包括异常，有一部分第一样本图像中不存在异常，对于负样本来说，异常类别监督标签用于表示负样本中不包括异常。在本申请实施例的下面描述中，以只采用正样本对异常分类子模型进行训练为例。
在一个实施例中，基于第一样本图像以及第一样本图像对应的异常类别监督标签对异常分类子模型进行训练的实施方式可以为：调用异常分类子模型对第一样本图像进行下采样异常分类处理，得到异常分类子模型输出的第一异常类别标签；基于第一异常类别标签和异常类别监督标签确定第一损失函数的值；按照减小所述第一损失函数的值的方向调整所述异常分类子模型的参数，直到所述第一损失函数的值达到最小，则确定当前的异常分类子模型是最优先的。可选的，此时所述的第一损失函数可以是交叉熵代价函数。
步骤S602、基于第二样本图像对应的异常类别训练标签以及第二样本图像，对应的第一定位图像对异常定位子模型进行训练。
其中，第二样本图像可以与第一样本图像相同，也可以与第一样本图像不同。在一个实施例中，第二样本图像对应的异常类别训练标签用于反映第二样本图像存在的异常类别，所述异常类别训练标签可以是调用步骤S601中训练完成的异常分类子模型进行下采样异常分类处理得到的。在其他实施例中，第二样本图像对应的异常类别训练标签也可以是调用其他能够进行异常识别的模型对第二样本图像进行异 常分类处理得到的。本申请实施例中，为了提高图像检测模型的准确性以及各个子模型之间的协作性，优选地基于步骤S601中训练完成的异常分类子模型得到第二样本图像对应的异常类别训练标签。
在一个实施例中，步骤S602的实施方式可以包括：
1)获取所述第二样本图像，并调用训练完成的异常分类子模型对所述第二样本图像进行下采样异常分类处理，得到所述异常类别训练标签和训练特征图像；
2)获取所述异常类别训练标签对应的第一类别标识；
3)调用所述初始定位模块，基于所述第一类别标识以及所述训练特征图像进行初步异常定位处理，得到所述第一定位图像；
4)调用所述异常定位子模型，对所述第一定位图像进行上采样异常定位处理，得到第二定位图像；
5)基于所述第二定位图像和所述第二样本图像，获取关于所述异常定位子模型对应的目标损失函数的值，并按照减小所述损失函数的值的方向，对所述异常定位子模型进行优化。
其中，第一定位图像和第二定位图像用于反映与异常类别训练标签关联的预测区域的属性信息。所述与异常类别训练标签关联的预测区域是指预测发生异常的异常区域。
其中，在步骤1)中所述训练特征图可以是异常分类子模型在对第二样本图像进行下采样异常分类过程中，最后一次下采样处理得到的图像。由前述可知，第一类别标识可以为异常类别号，服务器可以预先设置至少一组异常类别和异常类别标识之间的对应关系，当获取到异常类别训练标签时，确定异常类别训练标签所指示的异常类别，进一步的，在预先设置的对应关系中查找与确定出的异常对应的类别标识，将查找到的类别标识确定为第一类别标识。
在一个实施例中，在步骤3)中的具体实施方式可包括：获取与第一类别标识对应的第一特征权重向量；将所述第一特征权重向量与所述训练特征图像进行加权处理，得到第一定位图像。由前述可知，初始定位模块可以包括类别标识获取模块和特征过滤器，在步骤3)中，初始定位模块通过类别标识获取模块获取第一类别号，特征过滤器从训练特征图像获取与第一类别标识对应的第一特征权重向量。可选的， 所述获取与第一类别标识对应的第一特征权重向量的实施方式，与图2所示的实施例中获取与目标类别标识对应的目标特征权重向量的实施方式相同，具体可参见前述，此处不再赘述。
在一个实施例中，步骤5)中，所述基于所述第二定位图像和所述第二样本图像，获取所述异常定位子模型对应的目标损失函数的值，包括：基于所述第二定位图像包括的预测异常区域的属性信息和所述第二样本图像，得到第一优化图像和第二优化图像，并调用所述异常分类子模型，分别对所述第一优化图像和所述第二优化图像进行异常分类处理，得到第一概率和第二概率；获取所述预测异常区域在所述第二定位图像中所占的面积；获取对所述预测异常区域进行优化平滑的平滑函数的值；根据所述第一概率、所述第二概率、所述预测异常区域所占面积以及所述平滑函数的值，确定所述异常定位子模型对应的目标损失函数的值。
其中，目标损失函数的表达式可以如公式(1)所示：
Figure PCTCN2020115768-appb-000006
其中，X表示第二样本图像，m表示第二定位图像，
Figure PCTCN2020115768-appb-000007
表示第一概率，λ 1表示第一概率的权重值，第一概率用于表示将第二定位图像包括的预测异常区域和第二样本图像进行逐点相乘等操作后得到的第一优化图像，被异常分类子模型识别为存在异常区域的概率；
Figure PCTCN2020115768-appb-000008
表示第二概率，λ 2表示第二概率的权重值，从第二样本图像中除去与预测异常区域对应的区域得到第二优化图像，第二概率用于表示第二优化图像被异常分类子模型识别为存在异常区域的概率。
Figure PCTCN2020115768-appb-000009
表示预测异常区域在第二定位图像中的所占的面积，λ 3表示面积的权重值。应当理解的，通过异常定位子模型输出的第二定位图像可以是一张与第二样本图像大小相同的mask掩码。
参考图7a所示，为本申请实施例提供的一种第二样本图像、第一优化图像以及第二优化图像的示意图，其中71表示第二样本图像，72表示将第二样本图像包括的预测区域和第二样本图像进行逐点相乘等操作后得到的第一优化图像，第一优化图像中可保留了预测区域；73表示从第二样本图像中除去预测区域对应的区域后得到的第二优化图像，可见，第二优化图像中只去除了预测区域，保留了非预测区域。
在一个实施例中，可以通过以下公式(2)计算预测区域在第二定位图像中的所 占的面积：
Figure PCTCN2020115768-appb-000010
在公式(2)中，IxJ表示第二定位图像包括的总像素数，m(i,j)表示像素位置坐标。
在公式(1)中，
Figure PCTCN2020115768-appb-000011
表示对第二定位图像中的预测区域进行平滑的平滑函数，λ 4表示平滑函数的权重值，平滑函数的值可以通过下述公式(3)计算得到：
Figure PCTCN2020115768-appb-000012
基于上述描述以及公式(1)-(3)，本申请实施例提供了一种对目标定位子模块进行训练的结构示意图，参考图7b所示，图7b中示出确定第一概率和第二概率的过程。在图7b中，701表示异常分类子模型，702表示初始定位模块，703表示异常定位子模型。704表示第二样本图像，705表示经过异常分类子模型701、初始定位模块702以及异常定位子模型703进行处理后得到的目标定位图像。705中7051表示预测区域，将预测区域7051中的各个像素与第二样本图像704中的像素进行逐点相乘得到第一优化图像，调用异常分类子模型对第一优化图像进行下采样异常分类处理得到第一概率，在对异常定位子模型进行优化时，保证第一概率尽可能大。从第二样本图像中移除与预测异常区域对应的区域，得到第二优化图像，调用异常分类子模型对第二优化图像进行下采样异常分类处理得到第二概率，在对目标子模型进行优化时，保证第二概率尽可能大。
步骤S603、获取待检测图像，并调用图像检测模型中的异常分类子模型对所述待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像。
步骤S604、如果异常类别预测标签指示待检测图像中存在异常，则调用图像检测模型中的初始定位模块基于所述异常类别预测标签以及所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像。
为了节省服务器的功耗开销，在步骤S603之后，可首先检测异常类别预测标签是否表示待检测图像中存在异常；如果异常类别预测标签表示待检测图像中存在异常，则执行步骤S604；如果异常类别预测标签表示待检测图像中不存在，则可步骤S604，可直接输出异常类别预测标签。
步骤S605、调用图像检测模型中的异常定位子模型对所述初始定位图像进行上 采样异常定位处理，得到待检测图像对应的目标定位图像。
步骤S606、输出所述异常类别预测标签和所述目标定位图像。
在一个实施例中，目标定位图像可以是掩膜mask图像，mask图像中可显示预测区域，其他非预测区域被选定的掩膜mask覆盖。用户可以通过mask图像清新准确的确定出待检测图像中存在异常的目标区域。
在上述图像处理过程中，在异常类别预测标签指示待检测图像中包括目标异常区域时，在进行后续处理以得到目标定位图像，可以节省服务器的终端功耗。另外，可以通过调用异常分类子模型对待检测图像进行下采样异常处理得到异常类别预测标签，与现有技术中人为分析得到异常类别的方法相比，节省人力并且更加高效；进一步的，本申请实施例中，基于异常类别预测标签和下采样异常处理过程中获取到的目标特征图像进行初步异常定位处理，紧接着，再对初步异常处理得到的初始定位图像进行上采样异常定位处理，得到待检测图像中目标异常区域的目标定位图像，经过两次定位处理，可以保证得到的目标定位图像更加准确。
基于上述描述，本申请实施例还提供了一种医疗图像处理设备，参考图8a为本申请实施例提供的一种医疗图像处理设备的结构示意图，医疗处理设备可包括医疗图像采集器801、医疗图像分析器802以及医疗图像显示器803。
上述医疗图像分析器802对应前述实施例中的服务器，例如，图1a中所示的服务器103；所述医疗图像采集器801对应的前述实施例中的终端，例如，图1a中所示的图像采集终端120；医疗图像显示器803可以是终端，也可以是与终端连接的显示设备，例如，图1a中所示的图像显示设备140。所述医疗图像采集器801与医疗图像分析器802相连接，所述医疗图像分析器802与所述医疗图像显示器803相连接。
图8a所示的医疗图像处理设备可采集医疗图像并分析医疗图像，具体实现中，图8a中医疗图像采集器801、医疗图像分析器802以及医疗图像显示器803之间的交互可以如图8b所示，下面结合图8b具体介绍图8a所示的医疗图像设备采集医疗图像并分析医疗图像的过程，具体实现中可包括如下步骤：
步骤S81、医疗图像采集器获取医疗图像。
其中，此处所述医疗图像相当于图2和图6实施例中所述的待检测图像，所述医疗图像采集器可以内窥镜设备、CT设备等可以对人体进行检查的医疗设备。
步骤S82、医疗图像分析器对医疗图像进行下采样疾病分类处理，得到疾病类别预测标签和目标特征图像。
在一个实施例中，所述疾病类别预测标签相当于图2和图6实施例中所述异常类别预测标签。可选的，步骤S82包括的一些实施方式可参考图2和图6实施例中对待检测图像进行下采样异常分类处理，得到异常类别标签和目标特征图像的相关描述，在此不再赘述。
步骤S83、基于疾病类别预测标签以及目标特征图像进行初步疾病定位处理，得到医疗图像对应的初始定位图像。
在一个实施例中，疾病类别预测标签用于表示医疗图像中是否存在疾病，如果存在疾病，存在何种类型的疾病。如果疾病类别预测标签指示医疗图像不存在疾病，也即医疗图像中不存在目标病变区域，说明不需要进一步对医疗图像进行病变定位，则医疗图像分析器802可直接向医疗图像显示器803输出疾病类别预测标签；如果疾病类别预测标签指示医疗图像存在疾病，也即医疗图像中存在目标病变区域，可以进一步进行病变定位，则医疗图像分析器802基于疾病类别预测标签以及目标特征图像进行初步疾病定位处理，得到医疗图像对应的初始定位图像。
可选的，所述医疗图像分析器802在基于所述疾病类别预测标签以及目标特征图像进行初步疾病定位处理，得到所述初始定位图像时，执行如下操作：获取与所述疾病类别预测标签对应的目标类别标识；调用初始定位模块，获取所述目标类别标识对应的目标特征权重向量，并将所述目标特征权重向量与所述目标特征图像进行叠加处理，得到所述初始定位图像。
上述具体实施方式可以参考图2和图6实施例中，基于异常类别预测标签以及目标特征图像进行初始疾病定位处理，得到待检测图像对应的初始定位图像的一些描述，此处不再赘述。
步骤S84、医疗图像分析器对初始定位图像进行上采样疾病定位处理，得到医疗图像对应的目标定位图像。
在一个实施例，所述医疗图像分析器802在对所述医疗图像进行下采样疾病分 类处理时是调用疾病分类子模型(相当于前述的异常分类子模型)执行的，所述医疗图像分析器在对所述初始定位图像进行上采样疾病定位处理时是调用异常定位子模型执行的；所述疾病分类子模型包括至少一个第一采样模块，所述异常定位子模型包括至少一个第二采样模块，且第一采样模块的数量大于第二采样模块的数量。
所述医疗图像分析器在对所述医疗图像进行下采样疾病分类处理时，执行如下操作：获取疾病分类子模型，所述疾病分类子模型包括至少一个第一采样模块；调用所述至少一个第一采样模块，对所述医疗图像进行处理，提取出所述待检测图像包括的特征；基于提取到的特征进行异常检测，得到所述疾病类别预测标签；将所述至少一个第一采样模块中最后一个采样模块输出的特征图像，确定为所述目标特征图像。
所述医疗图像分析器在对所述初始定位图像进行上采样疾病定位处理时，执行如下操作：获取疾病定位子模型，所述疾病定位子模型包括至少一个第二采样模块，且第一采样模块的数量大于第二采样模块的数量；调用所述至少一个第二采样模块，对所述初始定位图像进行上采样疾病定位处理。
上述具体实施方式可以参考图2和图6实施例中对初始定位图像进行上采样异常定位处理得到待检测图像对应的目标定位图像的描述，此处不再赘述。
步骤S85、医疗图像显示器用于显示疾病类别预测标签和目标定位图像。
在一个实施例中，医疗图像分析器802得到目标定位图像之后，可以将目标定位图像和疾病类别标签输出给医疗图像显示器803，医疗图像显示器803在用户界面中显示疾病类别标签和目标定位图像。
可选的，医疗图像显示器803也可以与医疗图像采集器801连接，医疗图像采集器801采集到医疗图像之后，发送至医疗图像分析器802之前，可以将医疗图像发送至医疗图像显示器803，医疗图像显示器803在用户界面中显示医疗图像。
在一个实施例中，医疗图像显示器803还可以在用户界面中显示是否将医疗图像提交至医疗图像分析器802，以使得医疗图像分析器802对所述医疗图像进行疾病识别和定位处理的提示信息。如果医疗图像显示器803检测到针对提示信息的确认提交操作后，则医疗图像显示器803将医疗图像发送至医疗图像分析器802；如果医疗图像显示器803检测到针对提示信息的取消提交操作后，则医疗图像显示器803 可以不将医疗图像发送给医疗图像分析器802。
举例来说，假设医疗图像采集器801采集到医疗图像后，将医疗图像发送至医疗图像显示器803，医疗图像显示器803在用户界面上显示医疗图像以及是否将医疗图像提交至医疗图像分析器802的提示信息。
参考图8c所示，为本申请实施例提供的一种医疗显示器的用户界面的示意图，图8c中11表示医疗图像采集器801采集到的医疗图像，12表示提示信息，提示信息中可包括确认提交按钮120和取消提交按钮121；当用户点击确认提交按钮120时，医疗图像显示器803将医疗图像提交到医疗图像分析器802进行分析处理；当用户点击取消提交按钮121时，医疗图像显示器803则不将医疗图像提交至医疗图像分析器802。
在上述的图像检测过程中，可以通过对医疗图像进行下采样疾病分类处理得到疾病类别预测标签，与现有技术中人为分析得到疾病类别的方法相比，节省人力并且更加高效；进一步的，本申请实施例中，基于疾病类别预测标签和下采样处理过程中获取到的目标特征图像进行初步疾病定位处理；紧接着，再对初步异常处理得到的初始定位图像进行上采样疾病定位处理，得到医疗图像中医疗图像的目标定位图像，经过两次定位处理，可以保证得到的目标定位图像更加准确。
基于上述实施例描述，下面结合图9a-图9b以应用在医疗领域为例，具体描述一种医疗图像处理的应用场景。参考图9a所示，为本申请实施例提供的一种医疗图像处理的应用场景图，该应用场景中包括医疗图像采集器比如内窥镜器901，医疗图像显示器902以及医疗图像分析器903。假设医生使用内窥镜器901对人体消化道进行检查，获取到医疗视频数据；内窥镜器901可以将医疗视频数据发送至医疗显示器902，医疗图像显示器902通过脚本将医疗视频数据划分得到多帧医疗图像。
当检测到将医疗图像提交至医疗图像分析器903的操作后，医疗图像显示器902将多帧图像依次发送给医疗图像分析器903。参考图9b，为本申请实施例提供的一种医疗图像分析器903对医疗图像处理的示意图，医疗图像分析器903调用图像检测模型对医疗图像进行疾病类别识别和病变区域定位处理，得到疾病类别预测标签和病变区域的目标定位图像，如图9b中91所示。可选的，目标定位图像可包括mask 定位图像911和非mask定位图像912。进一步的，医疗图像分析器903将疾病类别预测标签和病变区域的目标定位图像发送给医疗图像显示器902，由医疗图像显示器902在用户界面中显示。医生可以根据医疗图像显示器902上显示的内容进行疾病诊断。
基于上述的图像处理方法，本申请实施例还提供了一种图像处理装置，参考图10为本申请实施例提供的图像处理装置的结构示意图，该图像处理装置可运行如下单元：
获取单元1001，用于获取待检测图像；
处理单元1002，用于对待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像；基于所述异常类别预测标签和所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像；对所述初始定位图像进行上采样异常定位处理，得到所述待检测图像对应的目标定位图像；
输出单元1003，用于输出所述异常类别预测标签和所述目标定位图像。其中，所述初始定位图像和所述目标定位图像用于反映所述异常类别预测标签关联的目标区域的属性信息。
在一个实施例中，所述处理单元1002用于，获取与所述异常类别预测标签对应的目标类别标识；调用初始定位模块，获取与所述目标类别标识对应的目标特征权重向量，并将所述目标特征权重向量与所述目标特征图进行叠加处理，得到所述初始定位图像。
在一个实施例中，所述处理单元1002用于，获取异常分类子模型，所述异常分类子模型包括至少一个第一采样模块；调用所述至少一个第一采样模块，对所述待检测图像进行处理，提取出所述待检测图像包括的特征；基于提取到的特征进行异常检测，得到所述异常类别预测标签；将所述至少一个第一采样模块中最后一个采样模块输出的特征图像，确定为所述目标特征图像。
在一个实施例中，所述处理单元1002用于，获取异常定位子模型，所述异常定位子模型包括至少一个第二采样模块，且第一采样模块的数量大于第二采样模块的数量；调用所述至少一个第二采样模块，对所述初始定位图像进行上采样异常定位 处理。
在一个实施例中，所述处理单元1002还用于，获取第二样本图像；调用所述异常分类子模型，对所述第二样本图像进行下采样异常分类处理，得到异常类别训练标签；调用所述初始定位模块，基于所述异常类别训练标签进行初步异常定位处理，得到第一定位图像；基于所述异常类别训练标签和所述第一定位图像，训练得到所述异常定位子模。
在一个实施例中，所述处理单元1002还用于，对所述第二样本图像进行下采样异常分类处理，得到训练特征图像；获取所述异常类别训练标签对应的第一类别标识；调用所述初始定位模块，基于所述第一类别标识以及所述训练特征图像进行初步异常定位处理，得到所述第一定位图像。
根据本申请的一个实施例，图10所示的图像处理装置中的各个单元可以分别或全部合并为一个或若干个另外的单元来构成，或者其中的某个(些)单元还可以再拆分为功能上更小的多个单元来构成，这可以实现同样的操作，而不影响本申请的实施例的技术效果的实现。上述单元是基于逻辑功能划分的，在实际应用中，一个单元的功能也可以由多个单元来实现，或者多个单元的功能由一个单元实现。在本申请的其它实施例中，基于图像处理装置也可以包括其它单元，在实际应用中，这些功能也可以由其它单元协助实现，并且可以由多个单元协作实现。
根据本申请的一个实施例，可以通过在包括中央处理单元(CPU)、随机存取存储介质(RAM)、只读存储介质(ROM)等处理元件和存储元件的例如计算机的通用计算设备上运行能够执行如图2以及图6中所示的相应方法所涉及的各步骤的计算机程序(包括程序代码)，来构造如图10中所示的图像处理装置设备，以及来实现本申请实施例的图像处理方法。所述计算机程序可以记载于例如计算机可读记录介质上，并通过计算机可读记录介质装载于上述计算设备中，并在其中运行。
基于上述方法的实施例以及装置的实施例，本申请实施例还提供了一种服务器，参考图11，为本申请实施例提供的一种服务器的结构示意图。图11所示的服务器可包括处理器1101和计算机存储介质1102。
计算机存储介质1102可以存储在终端的存储器中，所述计算机存储介质1102用于存储计算机程序，所述计算机程序包括程序指令，所述处理器1101用于执行所 述计算机存储介质1102存储的程序指令。处理器1101或称CPU(Central Processing Unit，中央处理器)是终端的计算核心以及控制核心，其适于实现一条或多条指令，具体适于加载并执行一条或多条指令从而实现相应方法流程或相应功能；在一个实施例中，处理器1101可以用于执行上述实施例中所述的图像处理方法。
本申请实施例还提供了一种计算机存储介质(Memory)，所述计算机存储介质是电子设备中的记忆设备，用于存放程序和数据。可以理解的是，此处的计算机存储介质既可以包括电子设备中的内置存储介质，当然也可以包括电子设备所支持的扩展存储介质。计算机存储介质提供存储空间，在该存储空间中存放了适于被处理器1101加载并执行的一条或多条指令，这些指令可以是一个或多个计算机程序(包括程序代码)。需要说明的是，此处的计算机存储介质可以是高速RAM存储器，也可以是非不稳定的存储器(non-volatile memory)，例如至少一个磁盘存储器；可选的还可以是至少一个位于远离前述处理器的计算机存储介质。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)或随机存储记忆体(Random Access Memory，RAM)等。
以上所揭露的仅为本申请部分实施例而已，当然不能以此来限定本申请之权利范围，因此依本申请权利要求所作的等同变化，仍属本申请所涵盖的范围。

Claims (23)

  1. 一种图像处理方法，其特征在于，由服务器执行，所述方法包括：
    获取待检测图像，并对所述待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像；
    基于所述异常类别预测标签以及所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像；
    对所述初始定位图像进行上采样异常定位处理，得到所述待检测图像对应的目标定位图像；及，
    输出所述异常类别预测标签和所述目标定位图像；
    其中，所述初始定位图像和所述目标定位图像用于反映所述异常类别预测标签关联的目标区域的属性信息。
  2. 如权利要求1所述的方法，其中，所述基于所述异常类别预测标签以及所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像，包括：
    获取与所述异常类别预测标签对应的目标类别标识；
    调用初始定位模块，获取与所述目标类别标识对应的目标特征权重向量，并将所述目标特征权重向量与所述目标特征图像进行叠加处理，得到所述初始定位图像。
  3. 如权利要求1所述的方法，其中，所述方法还包括：
    获取异常分类子模型，所述异常分类子模型包括至少一个第一采样模块；
    所述对所述待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像，包括：
    调用所述至少一个第一采样模块，对所述待检测图像进行处理，提取出所述待检测图像包括的特征；
    基于提取到的特征进行异常检测，得到所述异常类别预测标签；
    将所述至少一个第一采样模块中最后一个采样模块输出的特征图像，确定为所述目标特征图像。
  4. 如权利要求3所述的方法，其中，所述方法还包括：
    获取异常定位子模型，所述异常定位子模型包括至少一个第二采样模块，且第一采样模块的数量大于第二采样模块的数量；
    所述对所述初始定位图像进行上采样异常定位处理，得到所述待检测图像对应 的目标定位图像，包括：
    调用所述至少一个第二采样模块，对所述初始定位图像进行上采样异常定位处理。
  5. 如权利要求3所述的方法，其中，所述方法还包括：
    基于第一样本图像和所述第一样本图像对应的异常类别监督标签，训练得到所述异常分类子模型。
  6. 如权利要求4所述的方法，其中，所述方法还包括：
    获取第二样本图像；
    调用所述异常分类子模型，对所述第二样本图像进行下采样异常分类处理，得到异常类别训练标签；
    调用所述初始定位模块，基于所述异常类别训练标签进行初步异常定位处理，得到第一定位图像；
    基于所述异常类别训练标签和所述第一定位图像，训练得到所述异常定位子模型。
  7. 如权利要求6所述的方法，其中，所述方法还包括：
    对所述第二样本图像进行下采样异常分类处理，得到训练特征图像；
    所述调用所述初始定位模块，基于所述异常类别训练标签进行初步异常定位处理，得到第一定位图像，包括：
    获取所述异常类别训练标签对应的第一类别标识；
    调用所述初始定位模块，基于所述第一类别标识以及所述训练特征图像进行初步异常定位处理，得到所述第一定位图像。
  8. 如权利要求6所述的方法，其中，所述方法还包括：
    调用所述异常定位子模型，对所述第一定位图像进行上采样异常定位处理，得到所述第二样本图像对应的第二定位图像；
    基于所述第二定位图像和所述第二样本图像，获取所述异常定位子模型对应的目标损失函数的值，并按照减小所述目标损失函数的值的方向，对所述异常定位子模型进行优化；
    其中，所述第一定位图像和所述第二定位图像用于反映与所述异常类别训练标签关联的预测异常区域的属性信息。
  9. 如权利要求8所述的方法，其中，所述基于所述第二定位图像和所述第二样本图像，获取所述异常定位子模型对应的目标损失函数的值，包括：
    基于所述第二定位图像包括的预测异常区域的属性信息和所述第二样本图像，得到第一优化图像和第二优化图像，并调用所述异常分类子模型，分别对所述第一优化图像和所述第二优化图像进行异常分类处理，得到第一概率和第二概率；
    获取所述预测异常区域在所述第二定位图像中所占的面积；
    获取对所述预测异常区域进行优化平滑的平滑函数的值；
    根据所述第一概率、所述第二概率、所述预测异常区域所占面积以及所述平滑函数的值，确定所述异常定位子模型对应的目标损失函数的值。
  10. 如权利要求1所述的方法，其中，所述基于所述异常类别预测标签以及所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像之前，所述方法还包括：
    检测所述异常类别预测标签是否指示所述待检测图像中存在异常；
    如果所述异常类别预测标签指示所述待检测图像中存在异常，则执行所述基于所述异常类别预测标签以及所述目标特征图像进行初步异常定位处理的步骤。
  11. 如权利要求1所述的方法，其中，所述待检测图像包括医疗图像，所述异常类别预测标签包括疾病类别预测标签。
  12. 一种图像处理装置，其特征在于，包括：
    获取单元，用于获取待检测图像；
    处理单元，用于对待检测图像进行下采样异常分类处理，得到异常类别预测标签和目标特征图像；基于所述异常类别预测标签和所述目标特征图像进行初步异常定位处理，得到所述待检测图像对应的初始定位图像；对所述初始定位图像进行上采样异常定位处理，得到所述待检测图像对应的目标定位图像；及，
    输出单元，用于输出所述异常类别预测标签和所述目标定位图像；
    其中，所述初始定位图像和所述目标定位图像用于反映所述异常类别预测标签关联的目标区域的属性信息。
  13. 如权利要求12所述的装置，其中，所述处理单元用于，获取与所述异常类别预测标签对应的目标类别标识；调用初始定位模块，获取与所述目标类别标识对应的目标特征权重向量，并将所述目标特征权重向量与所述目标特征图像进行叠加处理，得到所述初始定位图像。
  14. 如权利要求12所述的装置，其中，所述处理单元用于，获取异常分类子模型，所述异常分类子模型包括至少一个第一采样模块；调用所述至少一个第一采样模块，对所述待检测图像进行处理，提取出所述待检测图像包括的特征；基于提取到的特征进行异常检测，得到所述异常类别预测标签；将所述至少一个第一采样模块中最后一个采样模块输出的特征图像，确定为所述目标特征图像。
  15. 如权利要求14所述的装置，其中，所述处理单元用于，获取异常定位子模型，所述异常定位子模型包括至少一个第二采样模块，且第一采样模块的数量大于第二采样模块的数量；调用所述至少一个第二采样模块，对所述初始定位图像进行上采样异常定位处理。
  16. 如权利要求15所述的装置，其中，所述处理单元还用于，获取第二样本图像；调用所述异常分类子模型，对所述第二样本图像进行下采样异常分类处理，得到异常类别训练标签；调用所述初始定位模块，基于所述异常类别训练标签进行初步异常定位处理，得到第一定位图像；基于所述异常类别训练标签和所述第一定位图像，训练得到所述异常定位子模型。
  17. 如权利要求16所述的装置，其中，所述处理单元还用于，对所述第二样本图像进行下采样异常分类处理，得到训练特征图像；获取所述异常类别训练标签对应的第一类别标识；调用所述初始定位模块，基于所述第一类别标识以及所述训练特征图像进行初步异常定位处理，得到所述第一定位图像。
  18. 一种服务器，其特征在于，所述服务器包括：
    处理器，适于实现一条或多条指令；以及，
    计算机存储介质，所述计算机存储介质存储有一条或多条指令，所述一条或多条指令适于由所述处理器加载并执行如权利要求1-11任一项所述的图像处理方法。
  19. 一种医疗图像处理设备，其特征在于，包括：医疗图像采集器、医疗图像分析器以及医疗图像显示器，所述医疗图像采集器与所述医疗图像分析器相连接，所述医疗图像显示器与所述医疗图像分析器相连；
    所述医疗图像采集器，用于获取医疗图像；
    所述医疗图像分析器，用于对所述医疗图像进行下采样疾病分类处理，得到疾病类别预测标签和目标特征图像；基于所述疾病类别预测标签以及目标特征图像进 行初步疾病定位处理，得到所述医疗图像对应的初始定位图像；对所述初始定位图像进行上采样疾病定位处理，得到所述医疗图像对应的目标定位图像；将所述疾病类别预测标签和所述目标定位图像输出至所述医疗图像显示器；
    所述医疗图像显示器，用于显示所述疾病类别预测标签和所述目标定位图像；
    其中，所述初始定位图像和所述目标定位图像用于反映所述疾病类别预测标签关联的目标区域的属性信息。
  20. 如权利要求19所述的医疗图像处理设备，其中，所述医疗图像分析器在基于所述疾病类别预测标签以及目标特征图像进行初步疾病定位处理，得到所述医疗图像对应的初始定位图像时，执行如下操作：
    获取所述疾病类别预测标签对应的目标类别标识；
    调用初始定位模块，获取与所述目标类别标识对应的目标特征权重向量，并将所述目标特征权重向量与所述目标特征图像进行叠加处理，得到所述初始定位图像。
  21. 如权利要求19所述的医疗图像处理设备，其中，所述医疗图像分析器在对所述医疗图像进行下采样疾病分类处理时，执行如下操作：
    获取疾病分类子模型，所述疾病分类子模型包括至少一个第一采样模块；
    调用所述至少一个第一采样模块，对所述医疗图像进行处理，提取出所述待检测图像包括的特征；
    基于提取到的特征进行异常检测，得到所述疾病类别预测标签；
    将所述至少一个第一采样模块中最后一个采样模块输出的特征图像，确定为所述目标特征图像。
  22. 如权利要求21所述的医疗图像处理设备，其中，所述医疗图像分析器在对所述初始定位图像进行上采样疾病定位处理时，执行如下操作：
    获取疾病定位子模型，所述疾病定位子模型包括至少一个第二采样模块，且第一采样模块的数量大于第二采样模块的数量；
    调用所述至少一个第二采样模块，对所述初始定位图像进行上采样疾病定位处理。
  23. 一种计算机存储介质，其特征在于，所述计算机存储介质中存储有计算机程序指令，所述计算机程序指令被处理器执行时，用于执行如权利要求1-11任一项所述的图像处理方法。
PCT/CN2020/115768 2019-11-12 2020-09-17 图像处理方法、装置、服务器、医疗图像处理设备及存储介质 Ceased WO2021093448A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP20886491.8A EP4006831A4 (en) 2019-11-12 2020-09-17 IMAGING METHOD AND DEVICE, SERVER, MEDICAL IMAGING DEVICE AND STORAGE MEDIA
US17/515,174 US12051199B2 (en) 2019-11-12 2021-10-29 Image processing method and apparatus, server, medical image processing device and storage medium

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911103557.7A CN110866908B (zh) 2019-11-12 2019-11-12 图像处理方法、装置、服务器及存储介质
CN201911103557.7 2019-11-12

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/515,174 Continuation US12051199B2 (en) 2019-11-12 2021-10-29 Image processing method and apparatus, server, medical image processing device and storage medium

Publications (1)

Publication Number Publication Date
WO2021093448A1 true WO2021093448A1 (zh) 2021-05-20

Family

ID=69653846

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/115768 Ceased WO2021093448A1 (zh) 2019-11-12 2020-09-17 图像处理方法、装置、服务器、医疗图像处理设备及存储介质

Country Status (4)

Country Link
US (1) US12051199B2 (zh)
EP (1) EP4006831A4 (zh)
CN (1) CN110866908B (zh)
WO (1) WO2021093448A1 (zh)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113420170A (zh) * 2021-07-15 2021-09-21 宜宾中星技术智能系统有限公司 大数据图像的多线程存储方法、装置、设备和介质
CN115018775A (zh) * 2022-05-24 2022-09-06 阿里巴巴（中国）有限公司 图像检测方法、装置、设备和存储介质
CN115190277A (zh) * 2022-09-08 2022-10-14 中达安股份有限公司 施工区域的安全监控方法、装置、设备及存储介质
CN116310346A (zh) * 2023-04-07 2023-06-23 平安科技（深圳）有限公司 图像分割方法、图像分割装置、电子设备及存储介质
CN116958128A (zh) * 2023-09-18 2023-10-27 中南大学 基于深度学习的医学图像自动定位方法
CN117018629A (zh) * 2022-10-13 2023-11-10 深圳市腾讯网络信息技术有限公司 数据处理方法、装置、设备、存储介质及程序产品
CN117132762A (zh) * 2023-08-29 2023-11-28 常州微亿智造科技有限公司 目标检测的优化方法、目标检测的优化装置

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110866908B (zh) 2019-11-12 2021-03-26 腾讯科技（深圳）有限公司 图像处理方法、装置、服务器及存储介质
CN111460991A (zh) * 2020-03-31 2020-07-28 科大讯飞股份有限公司 异常检测方法、相关设备及可读存储介质
CN111967538B (zh) * 2020-09-25 2024-03-15 北京康夫子健康技术有限公司 应用于小目标检测的特征融合方法、装置、设备以及存储介质
CN112966777B (zh) * 2021-03-26 2021-11-30 清华大学 一种基于人机交互的半自动标注方法及系统
CN113140292A (zh) * 2021-05-12 2021-07-20 平安国际智慧城市科技股份有限公司 图像异常区域浏览方法、装置、移动端设备及存储介质
US20220383128A1 (en) * 2021-05-28 2022-12-01 Zebra Technologies Corporation Image-based anomaly detection based on a machine learning analysis of an object
CN113793305A (zh) * 2021-08-23 2021-12-14 上海派影医疗科技有限公司 一种融合多信息的病理图像分类识别方法与系统
CN114332578B (zh) * 2021-09-15 2025-07-15 广州腾讯科技有限公司 图像异常检测模型训练方法、图像异常检测方法和装置
CN114387270B (zh) * 2022-03-22 2022-06-10 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机设备和存储介质
CN114464316B (zh) * 2022-04-11 2022-07-19 武汉大学 胃部异常风险等级预测方法、装置、终端及可读存储介质
CN114511749B (zh) * 2022-04-19 2022-06-28 武汉大学 图像处理方法、装置、计算机设备及存储介质
CN114842295B (zh) * 2022-04-29 2025-03-07 中铁第四勘察设计院集团有限公司 绝缘子故障检测模型的获得方法、装置及电子设备
CN114821328B (zh) * 2022-05-10 2025-09-09 南京南瑞信息通信科技有限公司 一种基于完全学习的电力图像处理方法及装置
CN115115563A (zh) * 2022-07-19 2022-09-27 上海商汤智能科技有限公司 数据处理方法、对象识别方法、装置、计算机设备及介质
CN115393627A (zh) * 2022-07-29 2022-11-25 上海东普信息科技有限公司 快件重量异常的识别方法、装置、设备及存储介质
CN115578370B (zh) * 2022-10-28 2023-05-09 深圳市铱硙医疗科技有限公司 一种基于脑影像的代谢区域异常检测方法及装置
CN116804958A (zh) * 2022-12-05 2023-09-26 慧之安信息技术股份有限公司 一种FreeSwitch的监控方法及装置
CN116245832B (zh) * 2023-01-30 2023-11-14 浙江医准智能科技有限公司 一种图像处理方法、装置、设备及存储介质
US20240257497A1 (en) * 2023-01-31 2024-08-01 Verily Life Sciences Llc Multi-frame analysis for classifying target features in medical videos
CN116071628B (zh) * 2023-02-06 2024-04-05 北京百度网讯科技有限公司 图像处理方法、装置、电子设备和存储介质
CN116524508B (zh) * 2023-04-19 2026-07-28 平安科技（深圳）有限公司 表格类图像的矫正方法及装置、存储介质、计算机设备
CN117152748B (zh) * 2023-07-24 2026-02-10 新石器慧通（北京）科技有限公司 一种3d目标检测模型的训练方法及3d目标检测方法
CN117094959B (zh) * 2023-08-11 2025-12-02 阿里巴巴达摩院(杭州)科技有限公司 图像处理方法、图像处理模型的训练方法
CN117475364A (zh) * 2023-10-25 2024-01-30 杭州阿里巴巴海外互联网产业有限公司 图片检测方法、设备及计算机存储介质
CN118367682B (zh) * 2024-06-19 2024-09-20 赛润电气(杭州)有限公司 一种电气设备智能状态识别调控方法及系统
CN118711002B (zh) * 2024-08-29 2025-01-17 阿里巴巴（中国）有限公司 期相识别模型的生成方法和图像的识别方法
CN119942224B (zh) * 2025-01-23 2025-08-19 无锡市惠山区人民医院 肠系膜上动脉异常自动检测方法及检测系统
CN119969936B (zh) * 2025-04-16 2025-11-11 云南省第一人民医院 鼻腔内窥镜驱动方法、电子设备及存储介质
CN121147670B (zh) * 2025-11-17 2026-04-10 深圳大学 视觉识别模型的训练方法、装置和计算机设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108665457A (zh) * 2018-05-16 2018-10-16 腾讯科技（深圳）有限公司 图像识别方法、装置、存储介质及计算机设备
CN109447960A (zh) * 2018-10-18 2019-03-08 神州数码医疗科技股份有限公司 一种对象识别方法和装置
US20190180153A1 (en) * 2015-08-14 2019-06-13 Elucid Bioimaging Inc. Methods and systems for utilizing quantitative imaging
CN110298831A (zh) * 2019-06-25 2019-10-01 暨南大学 一种基于分块深度学习的医学图像处理系统及其方法
CN110866908A (zh) * 2019-11-12 2020-03-06 腾讯科技（深圳）有限公司 图像处理方法、装置、服务器及存储介质

Family Cites Families (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6310967B1 (en) * 1998-04-29 2001-10-30 University Of South Florida Normal and abnormal tissue identification system and method for medical images such as digital mammograms
US8605981B2 (en) * 2010-11-05 2013-12-10 Cytognomix Inc. Centromere detector and method for determining radiation exposure from chromosome abnormalities
JP6076751B2 (ja) * 2013-01-22 2017-02-08 株式会社日立製作所 異常診断方法およびその装置
SG10201405182WA (en) * 2014-08-25 2016-03-30 Univ Singapore Technology & Design Method and system
WO2017090166A1 (ja) 2015-11-26 2017-06-01 オリンパス株式会社 画像処理装置、画像処理方法およびプログラム
US9773196B2 (en) * 2016-01-25 2017-09-26 Adobe Systems Incorporated Utilizing deep learning for automatic digital image segmentation and stylization
KR101901397B1 (ko) * 2016-12-06 2018-09-21 부산대학교 산학협력단 딥러닝을 이용한 수완부 방사선 영상으로부터 골 성숙도(ｓｍｉ) 자동 분석방법
CN110337644A (zh) * 2017-02-23 2019-10-15 谷歌有限责任公司 用于辅助病理学家标识放大的组织图像中的肿瘤细胞的方法和系统
WO2018222755A1 (en) * 2017-05-30 2018-12-06 Arterys Inc. Automated lesion detection, segmentation, and longitudinal identification
CN108009515B (zh) * 2017-12-14 2022-04-22 杭州远鉴信息科技有限公司 一种基于fcn的无人机航拍图像的输电线定位识别方法
US10679345B2 (en) * 2017-12-20 2020-06-09 International Business Machines Corporation Automatic contour annotation of medical images based on correlations with medical reports
TWI667996B (zh) * 2018-03-23 2019-08-11 中國醫藥大學附設醫院 乳房腫瘤輔助檢測模型及乳房腫瘤輔助檢測系統
CN108776787B (zh) * 2018-06-04 2020-09-29 京东数字科技控股有限公司 图像处理方法及装置、电子设备、存储介质
US10304193B1 (en) * 2018-08-17 2019-05-28 12 Sigma Technologies Image segmentation and object detection using fully convolutional neural network
CN109493317B (zh) * 2018-09-25 2020-07-07 哈尔滨理工大学 基于级联卷积神经网络的3d多椎骨分割方法
CN112789684B (zh) * 2018-10-01 2025-05-09 文塔纳医疗系统公司 用于预测对pd-1轴导向疗法的应答的方法和系统
US10803585B2 (en) * 2018-10-09 2020-10-13 General Electric Company System and method for assessing image quality
CN109410194B (zh) * 2018-10-19 2022-03-29 山东省计算中心（国家超级计算济南中心） 一种基于深度学习的食管癌病理图像处理方法
CN109522807B (zh) * 2018-10-22 2020-11-27 深圳先进技术研究院 基于自生成特征的卫星影像识别系统、方法及电子设备
US11011257B2 (en) * 2018-11-21 2021-05-18 Enlitic, Inc. Multi-label heat map display system
CN109598728B (zh) * 2018-11-30 2019-12-27 腾讯科技（深圳）有限公司 图像分割方法、装置、诊断系统及存储介质
CN109493954B (zh) * 2018-12-20 2021-10-19 广东工业大学 一种基于类别判别定位的sd-oct图像视网膜病变检测系统
CN109961019B (zh) * 2019-02-28 2021-03-26 华中科技大学 一种时空行为检测方法
WO2020202680A1 (ja) * 2019-03-29 2020-10-08 キヤノン株式会社 情報処理装置及び情報処理方法
CN110163121B (zh) * 2019-04-30 2023-09-05 腾讯科技（深圳）有限公司 图像处理方法、装置、计算机设备及存储介质
CN110490856B (zh) * 2019-05-06 2021-01-15 腾讯医疗健康(深圳)有限公司 医疗内窥镜图像的处理方法、系统、机器设备和介质
CN110110808B (zh) * 2019-05-16 2022-04-15 京东方科技集团股份有限公司 一种对图像进行目标标注的方法、装置及计算机记录介质
CN110390691B (zh) * 2019-06-12 2021-10-08 合肥合工安驰智能科技有限公司 一种基于深度学习的矿石尺度测量方法及应用系统
CN110363138B (zh) * 2019-07-12 2024-07-26 腾讯科技（深圳）有限公司 模型训练方法、图像处理方法、装置、终端及存储介质
TWI707663B (zh) * 2019-07-19 2020-10-21 財團法人資訊工業策進會 多視角乳房影像分析方法、多視角乳房影像分析系統及非暫態電腦可讀取媒體
CN114846507B (zh) * 2019-09-25 2025-10-28 阿斯泰刻有限公司 用于使用人工智能（ai）模型进行非侵入性基因检测的方法和系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190180153A1 (en) * 2015-08-14 2019-06-13 Elucid Bioimaging Inc. Methods and systems for utilizing quantitative imaging
CN108665457A (zh) * 2018-05-16 2018-10-16 腾讯科技（深圳）有限公司 图像识别方法、装置、存储介质及计算机设备
CN109447960A (zh) * 2018-10-18 2019-03-08 神州数码医疗科技股份有限公司 一种对象识别方法和装置
CN110298831A (zh) * 2019-06-25 2019-10-01 暨南大学 一种基于分块深度学习的医学图像处理系统及其方法
CN110866908A (zh) * 2019-11-12 2020-03-06 腾讯科技（深圳）有限公司 图像处理方法、装置、服务器及存储介质

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
SARIC MATKO; RUSSO MLADEN; STELLA MAJA; SIKORA MARJAN: "CNN-based Method for Lung Cancer Detection in Whole Slide Histopathology Images", 2019 4TH INTERNATIONAL CONFERENCE ON SMART AND SUSTAINABLE TECHNOLOGIES (SPLITECH), 1 August 2019 (2019-08-01), pages 1 - 4, XP033587567, DOI: 10.23919/SpliTech.2019.8783041 *
See also references of EP4006831A4

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113420170A (zh) * 2021-07-15 2021-09-21 宜宾中星技术智能系统有限公司 大数据图像的多线程存储方法、装置、设备和介质
CN113420170B (zh) * 2021-07-15 2023-04-14 宜宾中星技术智能系统有限公司 大数据图像的多线程存储方法、装置、设备和介质
CN115018775A (zh) * 2022-05-24 2022-09-06 阿里巴巴（中国）有限公司 图像检测方法、装置、设备和存储介质
CN115190277A (zh) * 2022-09-08 2022-10-14 中达安股份有限公司 施工区域的安全监控方法、装置、设备及存储介质
CN117018629A (zh) * 2022-10-13 2023-11-10 深圳市腾讯网络信息技术有限公司 数据处理方法、装置、设备、存储介质及程序产品
CN116310346A (zh) * 2023-04-07 2023-06-23 平安科技（深圳）有限公司 图像分割方法、图像分割装置、电子设备及存储介质
CN116310346B (zh) * 2023-04-07 2025-09-02 平安科技（深圳）有限公司 图像分割方法、图像分割装置、电子设备及存储介质
CN117132762A (zh) * 2023-08-29 2023-11-28 常州微亿智造科技有限公司 目标检测的优化方法、目标检测的优化装置
CN116958128A (zh) * 2023-09-18 2023-10-27 中南大学 基于深度学习的医学图像自动定位方法
CN116958128B (zh) * 2023-09-18 2023-12-26 中南大学 基于深度学习的医学图像自动定位方法

Also Published As

Publication number Publication date
EP4006831A1 (en) 2022-06-01
US20220051405A1 (en) 2022-02-17
CN110866908A (zh) 2020-03-06
EP4006831A4 (en) 2022-09-07
CN110866908B (zh) 2021-03-26
US12051199B2 (en) 2024-07-30

Similar Documents

Publication Publication Date Title
CN110866908B (zh) 图像处理方法、装置、服务器及存储介质
KR102155381B1 (ko) 인공지능 기반 기술의 의료영상분석을 이용한 자궁경부암 판단방법, 장치 및 소프트웨어 프로그램
US11244450B2 (en) Systems and methods utilizing artificial intelligence for placental assessment and examination
CN108230294B (zh) 图像检测方法、装置、电子设备和存储介质
WO2020207377A1 (zh) 一种图像识别模型训练及图像识别方法、装置及系统
CN110689025A (zh) 图像识别方法、装置、系统及内窥镜图像识别方法、装置
CN111325725A (zh) 视网膜图像识别方法及装置、电子设备、存储介质
US11721023B1 (en) Distinguishing a disease state from a non-disease state in an image
CN117975170B (zh) 基于大数据的医疗信息处理方法及系统
CN111144271B (zh) 一种内镜下自动识别活检部位及活检数量的方法及系统
CN116958539A (zh) 用于血管树分割的模型训练和图像处理方法、装置及设备
CN118675219A (zh) 基于眼底图像的糖尿病视网膜病变小病灶检测方法及系统
KR20210033902A (ko) 인공지능 기반 기술의 의료영상분석을 이용한 자궁경부암 진단방법, 장치 및 소프트웨어 프로그램
CN120318171A (zh) 一种利用宫颈细胞病理图像直接进行hpv预测的方法、装置、电子设备、存储介质及程序
CN115526834A (zh) 免疫荧光图像检测方法及装置、设备、存储介质
CN116958679B (zh) 一种基于弱监督的目标检测方法及相关设备
JP7603344B2 (ja) 組織検体画像の生成方法、及びこれを行うコンピューティングシステム
CN120783933A (zh) 一种内窥镜报告生成方法、装置、设备及可读存储介质
CN120672707A (zh) 内窥镜影像中息肉病变风险预测方法、系统及终端
CN111047582A (zh) 基于深度学习的小肠镜下克罗恩病辅助诊断系统
CN117636335A (zh) 一种基于改进YOLOv8与VGG19模型的内窥镜图像检测方法
JP7529592B2 (ja) 画像診断支援装置、画像診断支援方法、遠隔診断支援システム、ネット受託サービスシステム
CN118333942A (zh) 一种结直肠息肉检测方法、装置、设备及存储介质
CN113706449B (zh) 基于病理图像的细胞分析方法、装置、设备及存储介质
HK40021492B (zh) 图像处理方法、装置、服务器及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20886491

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2020886491

Country of ref document: EP

Effective date: 20220222

NENP Non-entry into the national phase

Ref country code: DE