KR102936070B1 - Data preprocessing method for drawing work - Google Patents

Abstract

According to one embodiment of the present disclosure, a method for processing an image using a neural network model is disclosed, the method being performed by one or more processors of a computing device.
The method may include the steps of: obtaining an image including at least one object; identifying a technical field to which the obtained image corresponds; performing object-level segmentation on the obtained image based on the identified technical field using a neural network model; and obtaining object-level segmented data on the obtained image.

Description

Data Preprocessing Method for Drawing Work

The present disclosure relates to a method for preprocessing data, and more particularly, to a method for preprocessing data for drawing inspection.

On-site, drawings are frequently revised for design changes, constructability improvements, material/component substitutions, permitting/licensing requirements, and customer requirements. However, in real-world construction sites, CAD native files, PDF snapshots, scans, and vector/raster files coexist, and layer/symbol systems differ by creator and vendor, making consistent comparison and verification difficult. Furthermore, differences in layer naming, color conventions, and symbol libraries across companies and departments can lead to different representations for identical objects, making existing simple shape comparisons (diffs) unable to identify semantic identities. Recent advances in computer vision, natural language processing, and shape recognition technologies have led to a growing demand for decomposing and understanding drawings into geometric, character, and semantic units, and for automatically matching front and back plates to classify and summarize changes. However, this necessitates preprocessing by segmenting drawings into semantic objects. However, during the preprocessing process of segmenting drawings into semantic objects, even though the objects and patterns that are important to consider differ across technical fields, segmenting objects in the same manner can lead to problems such as including unnecessary areas or missing important ones. Therefore, there is a growing need in the field to utilize neural network models to accurately distinguish between areas with technical significance, rather than simply "shape differences," depending on the technical field of the drawing. This approach allows for preprocessing drawings.

Meanwhile, while the present disclosure was derived at least based on the technical background discussed above, the technical task or purpose of the present disclosure is not limited to resolving the problems or shortcomings discussed above. That is, in addition to the technical issues discussed above, the present disclosure can cover various technical issues related to the content described below.

The present disclosure addresses the problem of preprocessing data for drawing verification. For example, the present disclosure resolves the problem of false detection/non-detection that occurs when simply comparing drawing modifications on a pixel basis by segmenting images into semantic objects for drawings used in an industrial environment and obtaining preprocessed data by segmenting the images into object units, and resolves the problem of false detection/non-detection that occurs when simply comparing drawing modifications on a pixel basis. The present disclosure also addresses the problem of decomposing and understanding into geometric, character, and semantic units, automatically aligning front and rear plates, and stably tracking the same object even when its position or order changes, thereby increasing the efficiency of drawing work.

Meanwhile, the technical task to be achieved by the present disclosure is not limited to the technical task mentioned above, and may include various technical tasks within a scope obvious to a person skilled in the art from the contents described below.

According to one embodiment of the present disclosure for achieving the aforementioned task, a method performed by a computing device is disclosed. The method may include the steps of: acquiring an image including at least one object; identifying a technology field to which the acquired image corresponds; performing object-level segmentation on the acquired image based on the identified technology field using a neural network model; and acquiring object-level segmented data on the acquired image.

In one embodiment, the step of obtaining an image including at least one object may include the step of obtaining an image including at least one of a title field, an annotation box, a component, or a drawing area.

In one embodiment, the step of identifying the technical field to which the acquired image corresponds may include the step of identifying the acquired image as at least one of the fields of architectural technology, electrical technology, and mechanical technology.

In one embodiment, the step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model may include the step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on preset criteria.

In one embodiment, the step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on the preset criteria may include the step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on at least one of a title field segmentation criterion; a dimension line and annotation separation criterion; or an internal region of interest (ROI) setting criterion.

In one embodiment, the step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model may include the step of setting a criterion for a region of interest (ROI) based on the identified technical field by utilizing the neural network model; and the step of performing object-level segmentation on the acquired image based on the criterion for the set region of interest (ROI).

In one embodiment, the step of determining the segmentation size of the image based on the object in the acquired image by utilizing the neural network model may include the step of determining the segmentation size of the image based on the density of the object in the acquired image by utilizing the neural network model.

In one embodiment, the step of determining the segmentation size of the image based on the density of objects in the acquired image by utilizing the neural network model may include at least one of the steps of determining the segmentation size of the image to be smaller than the average when the density of objects in the acquired image is higher than the average of the entire image; or the step of determining the segmentation size of the image to be larger than the average when the density of objects in the acquired image is lower than the average of the entire image.

In one embodiment, the step of acquiring segmented data in units of objects for the acquired image may further include the step of performing an evaluation on the segmented data in units of objects based on preset evaluation criteria; and the step of supplementing the segmented data in units of objects by utilizing the neural network model if the evaluation result does not pass the preset evaluation criteria.

In one embodiment, when the evaluation result does not pass the preset evaluation criteria, the step of supplementing the segmented data in the object unit by utilizing the neural network model may include a step of supplementing the segmented data in the object unit by utilizing the neural network model based on context information of the acquired image.

In one embodiment, the step of obtaining segmented data in units of objects for the acquired image may include a step of obtaining segmented data in units of objects by performing standardization on the acquired image in a format (Dictionary) in units of objects.

According to one embodiment of the present disclosure for achieving the above-described task, a computer program stored in a computer-readable storage medium is disclosed. When the computer program is executed on one or more processors, the computer program causes the one or more processors to perform operations for processing an image using a neural network model, the operations including: obtaining an image including at least one object; identifying a technical field to which the obtained image corresponds; performing object-level segmentation on the obtained image based on the identified technical field using the neural network model; and obtaining object-level segmented data on the obtained image.

In one embodiment, the act of obtaining an image including at least one object may include the act of obtaining an image including at least one of a title field, an annotation box, a part, or a drawing area.

In one embodiment, the operation of identifying the technical field to which the acquired image corresponds may include an operation of identifying the acquired image as at least one of the fields of architectural technology, electrical technology, and mechanical technology.

In one embodiment, the operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model may include an operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on preset criteria.

In one embodiment, the operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on the preset criteria may include an operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on at least one of a title field segmentation criterion; a dimension line and annotation separation criterion; or an internal region of interest (ROI) setting criterion.

In one embodiment, the operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model may include the operation of setting a criterion for a region of interest (ROI) based on the identified technical field by utilizing the neural network model; and the operation of performing object-level segmentation on the acquired image based on the criterion for the set region of interest (ROI).

In one embodiment, the operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model may include the operation of determining a segmentation size of the image based on an object in the acquired image by utilizing the neural network model; and the operation of performing object-level segmentation on the acquired image based on the determined segmentation size.

In one embodiment, the operation of determining the segmentation size of the image based on the object in the acquired image by utilizing the neural network model may include the operation of determining the segmentation size of the image based on the density of the object in the acquired image by utilizing the neural network model.

In one embodiment, the operation of determining the segmentation size of the image based on the density of objects in the acquired image by utilizing the neural network model may include at least one of: an operation of determining the segmentation size of the image to be smaller than the average when the density of objects in the acquired image is higher than the average of the entire image; or an operation of determining the segmentation size of the image to be larger than the average when the density of objects in the acquired image is lower than the average of the entire image.

In one embodiment, the operation of acquiring segmented data in units of objects for the acquired image may further include an operation of performing an evaluation on the segmented data in units of objects based on preset evaluation criteria; and an operation of supplementing the segmented data in units of objects by utilizing the neural network model if the evaluation result does not pass the preset evaluation criteria.

In one embodiment, when the evaluation result does not pass the preset evaluation criteria, the operation of supplementing the segmented data in the object unit by utilizing the neural network model may include an operation of supplementing the segmented data in the object unit by utilizing the neural network model based on context information of the acquired image.

A computing device according to one embodiment of the present disclosure for achieving the aforementioned task is disclosed. The device includes at least one processor, and the at least one processor may be configured to: acquire an image including at least one object; identify a technology field to which the acquired image pertains; perform object-level segmentation on the acquired image based on the identified technology field using a neural network model; and acquire object-level segmented data on the acquired image.

The present disclosure relates to a technique for preprocessing data for drawing inspection, and in particular, to a technique for overcoming the limitations of simple image-based pixel comparison by segmenting various types of drawings used in industrial settings into semantic objects, thereby enabling more precise and efficient drawing inspection. For example, conventional drawing inspection methods compare the entire image pixel by pixel to determine whether there is a change. Therefore, false positives occur, where changes that are not actual design changes are incorrectly recognized as changes due to differences in resolution, rendering methods, and noise generated during scanning. Conversely, false negatives occur frequently, where changes that are important to the actual design intent, such as changes in dimension lines or object properties, are not detected. These problems can lead to various side effects, such as quality degradation, design errors, and approval delays, in projects requiring large-scale design drawings or multidisciplinary collaboration. In contrast, the present disclosure decomposes and understands elements included in a drawing into geometric objects (lines, circles, polygons, etc.), text objects (dimensions, labels, part names, etc.), and semantic objects (roles of parts, electrical connections, mechanical functional elements, etc.), and preprocesses these into structured data (JSON, etc.) by object unit, thereby enabling detection of changes in meaningful units beyond simple pixel differences.

Meanwhile, the effects of the present disclosure are not limited to the effects mentioned above, and various effects may be included within a range apparent to those skilled in the art from the contents described below.

FIG. 1 is a block diagram of a computing device for processing an image using a neural network model according to one embodiment of the present disclosure.
FIG. 2 is a schematic diagram showing an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.
FIG. 3 is a flowchart illustrating a method for processing an image using a neural network model according to one embodiment of the present disclosure.
FIG. 4 is a schematic diagram illustrating a process of acquiring an image including at least one object and identifying a technical field to which the acquired image corresponds according to one embodiment of the present disclosure.
FIG. 5 is a schematic diagram illustrating a process of performing object-level segmentation on the acquired image based on the identified technical field by utilizing a neural network model according to one embodiment of the present disclosure.
FIG. 6 is a schematic diagram illustrating a process of determining a segmentation size of an image according to an object in an acquired image and performing object-level segmentation on the acquired image based on the determined segmentation size, using a neural network model according to one embodiment of the present disclosure.
FIG. 7 is a schematic diagram illustrating a process of supplementing the segmented data of the object unit by utilizing the neural network model based on context information of the image according to one embodiment of the present disclosure.
FIG. 8 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are provided to facilitate understanding of the present disclosure. However, it will be apparent that these embodiments may be practiced without these specific details.

As used herein, the terms "component," "module," "system," and the like refer to computer-related entities, hardware, firmware, software, a combination of software and hardware, or an execution of software. For example, a component may be, but is not limited to, a procedure running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device may be a component. One or more components may reside within a processor and/or a thread of execution. A component may be localized within a single computer. A component may be distributed between two or more computers. Furthermore, these components may execute from various computer-readable media having various data structures stored therein. Components may communicate via local and/or remote processes, for example, by signals comprising one or more data packets (e.g., data from one component interacting with another component in a local system, a distributed system, and/or data transmitted to another system via a network such as the Internet via signals).

Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from context, "X employs A or B" is intended to mean either of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, "X employs A or B" can apply to any of these cases. Furthermore, the term "and/or" as used herein should be understood to refer to and include all possible combinations of one or more of the associated items listed.

Additionally, the terms "comprises" and/or "comprising" should be understood to imply the presence of the features and/or components in question. However, it should be understood that the terms "comprises" and/or "comprising" do not exclude the presence or addition of one or more other features, components, and/or groups thereof. Furthermore, unless otherwise specified or clear from the context to refer to the singular form, the singular in the specification and claims should generally be construed to mean "one or more."

And, the term "at least one of A or B" should be interpreted to mean "if it includes only A", "if it includes only B", or "if it is combined in the composition of A and B".

Those skilled in the art should further appreciate that the various illustrative logical blocks, configurations, modules, circuits, means, logics, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application. However, such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The description of the disclosed embodiments is provided to enable a person skilled in the art to make or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art. The general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not limited to the embodiments disclosed herein. The present invention is to be construed in the widest scope consistent with the principles and novel features disclosed herein.

In the present disclosure, network function and artificial neural network and neural network can be used interchangeably.

FIG. 1 is a block diagram of a computing device for processing an image using a neural network model according to one embodiment of the present disclosure.

The configuration of the computing device (100) illustrated in FIG. 1 is merely a simplified example. In one embodiment of the present disclosure, the computing device (100) may include other configurations for performing the computing environment of the computing device (100), and only some of the disclosed configurations may constitute the computing device (100).

A computing device (100) may include a processor (110), memory (130), and network unit (150).

The processor (110) may be configured with one or more cores, and may include a processor for data analysis and deep learning, such as a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), and a tensor processing unit (TPU) of a computing device. The processor (110) may read a computer program stored in the memory (130) and perform data processing for machine learning according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the processor (110) may perform operations for learning a neural network model. The processor (110) may perform calculations for learning a neural network model, such as processing input data for learning in deep learning (DL), extracting features from input data, calculating errors, and updating weights of a neural network model using backpropagation. At least one of the CPU, GPGPU, and TPU of the processor (110) may process learning of the neural network model. For example, a CPU and a GPGPU can work together to train a neural network model and classify data using the neural network model. Furthermore, in one embodiment of the present disclosure, processors of multiple computing devices can be used together to train a neural network model and classify data using the neural network model. Furthermore, a computer program executed on a computing device according to one embodiment of the present disclosure may be a CPU, GPGPU, or TPU executable program.

According to one embodiment of the present disclosure, the memory (130) can store any form of information generated or determined by the processor (110) and any form of information received by the network unit (150).

According to one embodiment of the present disclosure, the memory (130) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The computing device (100) may also operate in relation to web storage that performs the storage function of the memory (130) on the internet. The description of the above-described memory is merely an example, and the present disclosure is not limited thereto.

The network unit (150) according to one embodiment of the present disclosure can use various wired communication systems such as a public switched telephone network (PSTN), xDSL (x Digital Subscriber Line), RADSL (Rate Adaptive DSL), MDSL (Multi Rate DSL), VDSL (Very High Speed DSL), UADSL (Universal Asymmetric DSL), HDSL (High Bit Rate DSL), and a local area network (LAN).

In addition, the network unit (150) presented in the present disclosure can use various wireless communication systems such as CDMA (Code Division Multi Access), TDMA (Time Division Multi Access), FDMA (Frequency Division Multi Access), OFDMA (Orthogonal Frequency Division Multi Access), SC-FDMA (Single Carrier-FDMA) and other systems.

In the present disclosure, the network unit (150) may be configured regardless of the communication mode, such as wired or wireless, and may be configured as various communication networks, such as a personal area network (PAN) and a wide area network (WAN). In addition, the network may be the well-known World Wide Web (WWW), and may also utilize a wireless transmission technology used for short-distance communication, such as infrared (IrDA: Infrared Data Association) or Bluetooth. The technologies described in the present disclosure may also be used in other networks mentioned above.

FIG. 2 illustrates an exemplary structure of an artificial intelligence-based model according to one embodiment of the present disclosure.

Throughout this specification, the terms artificial intelligence model, artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably.

A neural network can be composed of a set of interconnected computational units, generally referred to as nodes. These nodes can also be referred to as neurons. A neural network consists of at least one node. The nodes (or neurons) that make up a neural network can be interconnected by one or more links.

Within a neural network, one or more nodes connected via links can form a relationship between input nodes and output nodes. The concept of input nodes and output nodes is relative, meaning that any node that is in an output node relationship with one node can also be in an input node relationship with another node, and vice versa. As described above, the relationship between input nodes and output nodes can be created based on links. One input node can be connected to one or more output nodes via links, and vice versa.

In a relationship between input nodes and output nodes connected through a single link, the data of the output node can have its value determined based on the data input to the input node. Here, the link interconnecting the input nodes and output nodes can have a weight. The weight can be variable and can be varied by the user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node through each link, the output node can determine the output node value based on the values input to the input nodes connected to the output node and the weight set on the link corresponding to each input node.

As described above, a neural network is a network in which one or more nodes are interconnected through one or more links, forming input and output node relationships within the network. The characteristics of a neural network can be determined based on the number of nodes and links within the network, the relationships between the nodes and links, and the weights assigned to each link. For example, if two neural networks have the same number of nodes and links but different weight values for the links, the two neural networks can be perceived as different from each other.

A neural network can be composed of a set of one or more nodes. A subset of the nodes comprising the neural network can form a layer. Some of the nodes comprising the neural network can form a layer based on their distances from the initial input node. For example, a set of nodes that are n distances from the initial input node can form n layers. The distance from the initial input node can be defined by the minimum number of links required to reach the node from the initial input node. However, this definition of a layer is arbitrary for illustrative purposes, and the degree of a layer within a neural network can be defined in a different way than described above. For example, a layer of nodes can be defined by its distance from the final output node.

In one embodiment of the present disclosure, a set of neurons or nodes may be defined as a layer.

An initial input node may refer to one or more nodes within a neural network into which data is directly input without going through links with other nodes. Alternatively, within a neural network, it may refer to nodes that do not have other input nodes connected by links in the relationship between nodes based on links. Similarly, a final output node may refer to one or more nodes within a neural network that do not have output nodes in their relationship with other nodes. Furthermore, a hidden node may refer to nodes that constitute a neural network other than the initial input node and the final output node.

A neural network according to one embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be the same as the number of nodes in an output layer, and the number of nodes decreases and then increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be less than the number of nodes in an output layer, and the number of nodes increases as it progresses from the input layer to the hidden layer. In addition, a neural network according to another embodiment of the present disclosure may be a neural network in which the number of nodes in an input layer may be greater than the number of nodes in an output layer, and the number of nodes decreases as it progresses from the input layer to the hidden layer. A neural network according to another embodiment of the present disclosure may be a neural network in the form of a combination of the above-described neural networks.

An AI-based model according to one embodiment of the present disclosure may include a deep neural network (DNN). A DNN may refer to a neural network that includes multiple hidden layers in addition to an input layer and an output layer. Using a DNN, it is possible to identify latent structures in data. That is, the latent structures of a photo, text, video, voice, protein sequence structure, gene sequence structure, peptide sequence structure, music (e.g., what objects are in a photo, what the content and emotion of a text are, what the content and emotion of a voice are, etc.), and/or the binding affinity between a peptide and MHC can be identified. Deep neural networks may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), autoencoders, restricted Boltzmann machines (RBMs), deep belief networks (DBNs), Q-networks, U-networks, Siamese networks, generative adversarial networks (GANs), transformers, and the like. The description of the above-described deep neural networks is merely an example and the present disclosure is not limited thereto.

The artificial intelligence-based model of the present disclosure can be represented by a network structure of any structure described above, including an input layer, a hidden layer, and an output layer.

The neural network that can be used in the artificial intelligence-based model of the present disclosure may be trained using at least one of supervised learning, unsupervised learning, semi-supervised learning, transfer learning, active learning, or reinforcement learning. Training of the neural network may be a process of applying knowledge to the neural network to perform a specific action.

Neural networks can be trained to minimize output errors. This process involves repeatedly inputting training data into the neural network, calculating the neural network output and target error for the training data, and backpropagating the neural network error from the output layer to the input layer to update the weights of each node in the neural network to reduce the error. In supervised learning, training data with the correct answer for each training data is used (i.e., labeled training data). In unsupervised learning, the correct answer may not be labeled for each training data. For example, in supervised learning for data classification, the training data may be data with each category labeled. Labeled training data is input to the neural network, and the error can be calculated by comparing the output (category) of the neural network with the training data labels. Alternatively, in unsupervised learning for data classification, the error can be calculated by comparing the input training data with the neural network output. The calculated error is backpropagated in the neural network in the backward direction (i.e., from the output layer to the input layer), and the connection weights of each node in each layer of the neural network can be updated according to the backpropagation. The amount of change in the connection weights of each node to be updated can be determined by the learning rate. The neural network's calculation of the input data and the backpropagation of the error can constitute a learning cycle (epoch). The learning rate can be applied differently depending on the number of iterations of the neural network's learning cycle. For example, a high learning rate can be used in the early stages of neural network training to quickly achieve a certain level of performance, thereby increasing efficiency. A lower learning rate can be used in the later stages of training to increase accuracy.

In neural network training, training data can typically be a subset of real-world data (i.e., the data to be processed using the trained neural network). Therefore, there can be a learning cycle where errors on the training data decrease but errors on the real-world data increase. Overfitting is a phenomenon where excessive training on the training data leads to increased errors on the real-world data. For example, a neural network trained on yellow cats may fail to recognize cats when shown non-yellow colors, a type of overfitting. Overfitting can increase errors in machine learning algorithms. Various optimization methods can be used to prevent overfitting. These methods include increasing the training data, regularization, dropout, which disables some nodes in the network during the learning process, and the use of batch normalization layers.

A computer-readable medium storing a data structure according to one embodiment of the present disclosure is disclosed. The aforementioned data structure can be stored in a storage unit in the present disclosure, executed by a processor, and transmitted and received by a communication unit.

A data structure can refer to the organization, management, and storage of data that enables efficient access and modification. A data structure can refer to the organization of data to solve specific problems (e.g., data analysis, data retrieval, data storage, data modification). A data structure can also be defined as the physical or logical relationships between data elements designed to support specific data processing functions. Logical relationships between data elements can include connections between user-defined data elements. Physical relationships between data elements can include actual relationships between data elements physically stored on a computer-readable storage medium (e.g., persistent storage). Specifically, a data structure can include a collection of data, relationships between data, and functions or commands applicable to the data. An effectively designed data structure allows a computing device to perform operations while minimizing the use of its resources. Specifically, a computing device can improve the efficiency of operations, reading, inserting, deleting, comparing, exchanging, and searching through an effectively designed data structure.

Data structures can be categorized as linear or nonlinear, depending on their form. A linear data structure can be a structure in which only one data item is linked to the next. Linear data structures can include lists, stacks, queues, and deques. A list can refer to a series of data sets with an internal order. Lists can also include linked lists. A linked list is a data structure in which data is linked in a single line, each item having a pointer. In a linked list, a pointer can contain information about the next or previous item. Linked lists can be expressed as singly linked lists, doubly linked lists, or circular linked lists, depending on their form. A stack can be a data listing structure with limited data access. A stack can be a linear data structure in which data operations (e.g., insertion or deletion) can only be performed at one end of the data structure. Data stored in a stack can be a Last-in-First-out (LIFO) data structure. A queue is a data structure with limited access to data. Unlike a stack, it can be a first-in, first-out (FIFO) data structure, with later data being retrieved later. A deck can be a data structure that can process data at both ends.

A nonlinear data structure can be a structure in which multiple pieces of data are connected behind a single piece of data. Nonlinear data structures can include graph data structures. A graph data structure can be defined by vertices and edges, and an edge can include a line connecting two different vertices. Graph data structures can include tree data structures. A tree data structure can be a data structure in which there is only one path connecting two different vertices among multiple vertices included in the tree. In other words, it can be a data structure that does not form a loop in a graph data structure.

Throughout this specification, the terms artificial intelligence-based model, computational model, neural network, network function, and neural network may be used interchangeably. Hereinafter, they are collectively referred to as neural networks. A data structure may include a neural network. And, a data structure including a neural network may be stored on a computer-readable medium. A data structure including a neural network may also include preprocessed data for processing by a neural network, data input to a neural network, neural network weights, neural network hyperparameters, data obtained from a neural network, activation functions associated with each node or layer of a neural network, loss functions for neural network learning, etc. A data structure including a neural network may include any of the components disclosed above. That is, a data structure including a neural network may be configured to include all or any combination of preprocessed data for processing by a neural network, data input to a neural network, neural network weights, neural network hyperparameters, data obtained from a neural network, activation functions associated with each node or layer of a neural network, loss functions for neural network learning, etc. In addition to the aforementioned configurations, a data structure including a neural network may include any other information that determines the characteristics of the neural network. Furthermore, the data structure may include any form of data used or generated in the computational process of the neural network, and is not limited to the aforementioned. The computer-readable medium may include a computer-readable recording medium and/or a computer-readable transmission medium. A neural network may be composed of a set of interconnected computational units, which may generally be referred to as nodes. These nodes may also be referred to as neurons. A neural network is composed of at least one node.

The data structure may include data input to a neural network. The data structure including the data input to the neural network may be stored on a computer-readable medium. The data input to the neural network may include training data input during the neural network training process and/or input data input to the neural network after training has been completed. The data input to the neural network may include data that has undergone preprocessing and/or data that is the target of preprocessing. Preprocessing may include a data processing process for inputting data to the neural network. Accordingly, the data structure may include data that is the target of preprocessing and data generated by the preprocessing. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

The data structure may include weights of the neural network. (In this specification, the terms "weight" and "parameter" may be used interchangeably.) And the data structure including the weights of the neural network may be stored in a computer-readable medium. The neural network may include a plurality of weights. The weights may be variable and may be varied by a user or an algorithm so that the neural network can perform a desired function. For example, when one or more input nodes are interconnected to one output node by respective links, the output node may determine a data value output from the output node based on the values input to the input nodes connected to the output node and the weights set for the links corresponding to each input node. The above-described data structure is merely an example, and the present disclosure is not limited thereto.

By way of example and not limitation, the weights may include weights that vary during the neural network training process and/or weights that have completed neural network training. The weights that vary during the neural network training process may include weights at the start of the training cycle and/or weights that vary during the training cycle. The weights that have completed neural network training may include weights that have completed the training cycle. Accordingly, a data structure including the weights of a neural network may include a data structure including weights that vary during the neural network training process and/or weights that have completed neural network training. Therefore, the above-described weights and/or combinations of each weight are included in the data structure including the weights of a neural network. The above-described data structures are merely examples and the present disclosure is not limited thereto.

A data structure including neural network weights can be stored in a computer-readable storage medium (e.g., memory, hard disk) after going through a serialization process. Serialization can be a process of converting a data structure into a form that can be stored on the same or different computing devices and later reconstructed and used. A computing device can serialize the data structure to transmit and receive data over a network. The serialized data structure including neural network weights can be reconstructed on the same computing device or another computing device through deserialization. The data structure including neural network weights is not limited to serialization. Furthermore, the data structure including neural network weights can include a data structure that increases computational efficiency while minimizing the use of computing device resources (e.g., a B-Tree, an R-Tree, a Trie, an m-way search tree, an AVL tree, a Red-Black Tree in nonlinear data structures). The foregoing is merely an example, and the present disclosure is not limited thereto.

The data structure may include hyperparameters of a neural network. Furthermore, the data structure including the hyperparameters of the neural network may be stored on a computer-readable medium. The hyperparameters may be variables that can be varied by the user. The hyperparameters may include, for example, a learning rate, a cost function, the number of learning cycle repetitions, weight initialization (e.g., setting a range of weight values to be subject to weight initialization), and the number of hidden units (e.g., the number of hidden layers, the number of nodes in the hidden layer). The above-described data structure is merely an example, and the present disclosure is not limited thereto.

An AI-based model according to one embodiment of the present disclosure may include a Large Language Model (LLM). The term "large language model" in the present disclosure may refer to an AI-based model trained using a large amount of training data to perform natural language processing (NLP). The large language model may include a transformer, a model of the encoder family of transformers, and/or a model of the decoder family of transformers. A model of the encoder family of transformers may correspond to an AI model that uses the encoder structure of a transformer. A model of the decoder family of transformers may correspond to an AI model that uses the decoder structure of a transformer.

In one embodiment, a transformer may be composed of an encoder that encodes input data and a decoder that decodes the encoded data. The transformer may have a structure that takes a series of input data as input, performs encoding and decoding steps, and outputs a series of output data. In one embodiment, the series of input data may be processed into a form that the transformer can compute. The process of processing the series of input data into a form that the transformer can compute may include a tokenizing process and an embedding process. The tokenizing process may refer to a process of dividing a series of input data into tokens of a certain unit. For example, the unit may include a word unit. The embedding process may refer to a process of converting at least one token tokenized from a series of input data into an embedding vector.

In one embodiment, the transformer can obtain an embedding vector to be input to the encoder by combining a token embedding vector embedding at least one token corresponding to a series of input data, a segment embedding vector distinguishing a sentence containing each token, and a position embedding vector reflecting the position of the token. Models in the encoder series and models in the decoder series of the transformer can also obtain an embedding vector by performing the same method.

In one embodiment, the encoder and decoder within the transformer may utilize an attention algorithm to encode and decode a series of input data. The attention algorithm may refer to an algorithm that, for a given query, calculates a similarity by applying a softmax function to the attention score obtained by multiplying the query by a key by a matrix, and then calculates an attention value for the query by multiplying the calculated similarity by a value by a matrix.

In one embodiment, a self-attention algorithm may refer to an attention algorithm that utilizes queries, keys, and values generated by multiplying a query weight, a key weight, and a value weight respectively by the same embedding vector. A cross-attention algorithm may refer to an attention algorithm that utilizes a query generated by multiplying a first embedding vector by a query weight, and a key and value generated by multiplying a second embedding vector by a key weight and a value weight respectively. The query weight, key weight, and value weight may be trainable parameters that are updated through a large-scale language model learning process.

In one embodiment, the encoder of the transformer may include an embedding layer, a self-attention layer that applies a self-attention algorithm to the embedding vector, a normalization layer, and a feed-forward neural network (FFN). In addition, the encoder may have a form in which N unit structures including the self-attention layer, the normalization layer, and the feed-forward neural network are connected. The decoder of the transformer may include an embedding layer, a masked self-attention layer, a normalization layer, a cross-attention layer that applies a cross-attention algorithm, and a feed-forward neural network. In addition, the decoder may have a form in which N unit structures including the masked self-attention layer, the normalization layer, the cross-attention layer, and the feed-forward neural network are connected. The masked self-attention layer may correspond to a layer that obtains an attention value for each of sequences sequentially including words among a plurality of words included in a series of input data.

Transformers may include additional components such as linear layers and softmax layers in addition to encoders and decoders. Transformer encoder and decoder models may also include the aforementioned additional components in addition to encoders and decoders, respectively. Methods for constructing transformers using attention algorithms may include methods disclosed in Vaswani et al., Attention Is All You Need, 2017 NIPS, which is incorporated herein by reference.

In one embodiment, an attention layer such as a self-attention layer, a masked self-attention layer, or a cross-attention layer may correspond to a multi-head attention layer that includes multiple attention layers in parallel. The multi-head attention layer may concatenate attention values output from each of the multiple attention layers into a matrix, and output an output attention value by multiplying the output weight by the concatenated matrix. The output attention value output from the multi-head attention layer may have the same size as an attention value output from a single attention layer.

In one embodiment, the Transformer may be trained through a Masked Language Model (MLM) process, a Next Sentence Prediction (NSP) process, or the like. The MLM process may refer to a learning process that predicts masked words from a set of training data in which some words are masked. The NSP process may refer to a learning process that determines whether two sentences are connected sentences from a set of training data containing arbitrary sentences.

In one embodiment, a large-scale language model can process various data formats, such as natural language text, image data, audio data, and video data. The large-scale language model can embed data in order to convert data of various data formats into a computable series of data. The large-scale language model can process additional data representing the relative positional relationship or topological relationship between a series of input data. Alternatively, vectors representing the relative positional relationship or topological relationship between the input data can be additionally reflected in the series of input data to embed the series of input data. In one example, the relative positional relationship between the series of input data may include, but is not limited to, word order within a natural language sentence, the relative positional relationship between each segmented image, and the temporal order of segmented audio waveforms. The process of adding information representing the relative positional relationship or topological relationship between a series of input data may be referred to as positional encoding.

An example of a large-scale language model for processing image data (Vision Transformer, ViT) is disclosed in Dosovitskiy, et al., AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE, which is incorporated herein by reference.

An artificial intelligence model according to one embodiment of the present disclosure may include a multimodal large-scale language model. A multimodal large-scale language model may refer to a large-scale language model capable of understanding and processing relationships between different data formats, such as natural language text data, image data, audio data, and video data. The multimodal language model may include a plurality of encoders that encode input data corresponding to each data format. The multimodal language model may be trained to calculate similarity between embedding vectors encoded from encoders of each data format through training data including data of different data formats, and to calculate higher similarity for identical pairs and lower similarity for different pairs.

An example of a multimodal large-scale language model that understands and processes relationships between image data and natural language text data (Contrastive Language-Image Pre-training, CLIP) is described in Alec Radford, et al., LEARNING TRANSFERABLE VISUAL MODELS FROM NATURAL LANGUAGE SUPERVISION, which is incorporated herein by reference.

FIG. 3 is a flowchart illustrating a method for processing an image using a neural network model according to one embodiment of the present disclosure.

A computing device (100) according to one embodiment of the present disclosure may directly acquire "information for processing an image using a neural network model" or receive it from an external system. The external system may be a server, database, or the like that stores and manages information for processing an image using a neural network model. The computing device (100) may use the information acquired directly or received from the external system as "input data for processing an image using a neural network model."

According to one embodiment of the present disclosure, a computing device (100) may acquire an image including at least one object (S110). At this time, the image including at least one object may include a technical drawing, and may include, for example, architectural/civil engineering CAD, layout, P&ID, electrical/PCB, mold/machining, patent drawings, scanned paper drawings (PDF/image), etc., but is not limited thereto. In addition, the image including at least one object may include at least one of a title field, annotation box, a part, or a drawing area. At this time, the title field may mean metadata for tracking the identity, management, and version of the drawing, including the drawing number, drawing title, author, reviewer, approver name and signature, creation date, revision history, scale, drawing size (Size, A3, A4, etc.), company/organization name, and logo indicated at the lower right of the drawing (standardized by international standards ISO, KS, ANSI, etc.). In addition, the above-mentioned annotation box may refer to a text box in which a description of a specific part (e.g., “Dimensional unit is mm”, “Tolerance ±0.1 mm”), compliance with regulations/standards, references, or related drawing numbers are written to prevent the drawing interpreter from misunderstanding the design intent. Additionally, the above-mentioned part may refer to an actual object expressed as a shape (e.g., a line, a circle, a curve, etc.), and the above-mentioned drawing area may refer to the entire area on which the drawing is expressed on paper (or screen), but is not limited thereto. Meanwhile, the computing device (100) may identify a technical field to which the acquired image corresponds, and a description related thereto will be described later.

According to one embodiment of the present disclosure, the computing device (100) can identify the technical field to which the image acquired through step S110 corresponds (S120). For example, the computing device (100) can identify the acquired image as at least one of the fields of architectural technology, electrical technology, and mechanical technology. Specifically, the computing device (100) can identify the acquired image as the field of architectural technology if at least one of hatching, wall/slab/door/window symbols, scale bar, scale, room name (“Living, Bath, Corridor”), and plan/elevation/cross-section multi-view exists in the acquired image. However, the present invention is not limited to the above examples and various examples may be utilized. Meanwhile, for example, a PCB drawing is important for fine wiring width. If the computing device (100) applies the same ROI (large area centered) as the architectural standard in the process of performing object-level segmentation on the image to be described later, detailed wiring changes may be missed, resulting in inaccurate results. Accordingly, through one embodiment of the present disclosure, the computing device (100) can first identify the technical field of the acquired image, thereby accurately distinguishing parts with technical significance by reflecting the characteristics of each technical field, and utilizing this in the process of preprocessing the drawing. Meanwhile, a detailed description of the process by which the computing device (100) identifies the technical field to which the acquired image pertains will be described below with reference to FIG. 4.

According to one embodiment of the present disclosure, the computing device (100) may perform object-level segmentation on the acquired image based on the technology field identified through step S120 by utilizing a neural network model (S130). At this time, a pre-trained transformer (GPT series) model may be used as the neural network model, and examples thereof include Google's Gemini and Chat-GPT, but are not limited thereto, and various pre-trained neural network models may be used. For example, the computing device (100) may perform object-level segmentation on the acquired image based on the identified technology field by utilizing the neural network model based on preset criteria. At this time, the preset criteria may include at least one of a title section segmentation criterion, a dimension line and annotation separation criterion, or an internal region of interest (ROI) setting criterion. Specifically, the computing device (100) can set the criteria for segmenting the title block so that segmentation is performed on an object basis within a certain ratio (e.g., bottom 15%, right 30%) of the sheet size (A0, A1, A2, A3, etc.) for the title block, which is mostly arranged on the lower right side of the drawing according to international standards (ISO, KS, ANSI, etc.). As another example, the computing device (100) can obtain the criteria for setting the internal focus area (ROI), such as an example of “selecting an area where objects such as lines, circles, symbols, and text are gathered at a certain density (0.9) or more as an ROI” so as to include “an area where design objects are densely concentrated” for the internal focus area (ROI) and exclude “ancillary information such as a title block and annotations.” However, in addition to the above examples, various examples can be utilized for the preset criteria.

According to one embodiment of the present disclosure, the computing device (100) may utilize a neural network model to set criteria for a region of interest (ROI) according to the identified technical field, and may perform object-level segmentation of the acquired image based on the criteria for the set region of interest (ROI). For example, the computing device (100) may automatically segment room-level areas into ROIs in the architecture/civil engineering field, where structures, safety, and spatial layout are important, and may preferentially set ROI areas for objects that meet building code standards, such as “stair step height, doorway width,” where compliance with regulations is important, and may set criteria for the region of interest (ROI) by reinforcing ROI areas around load-bearing members (columns, beams) with high structural importance compared to the average of the entire drawing. However, in addition to the above examples, various examples may be utilized in the process of setting criteria for the region of interest (ROI) according to the identified technical field. Through this, the computing device (100) can preprocess the image by reducing unnecessary operations by performing segmentation on an object basis on the acquired image through criteria for key objects by field and regions of interest (ROI) set for the purpose of review, and accurately identifying technically meaningful areas in the actual technical field.

According to another embodiment, the computing device (100) may determine a segmentation size of the image according to an object in the acquired image, and perform object-by-object segmentation on the acquired image based on the determined segmentation size. For example, the computing device (100) may determine a segmentation size of the image based on the density of objects in the acquired image by utilizing the neural network model. More specifically, the computing device (100) may determine a segmentation size of the image to be smaller than the average when the density of objects in the acquired image is higher than the average of the entire image, or may determine a segmentation size of the image to be larger than the average when the density of objects in the acquired image is lower than the average of the entire image, thereby determining the segmentation size of the image. Through this, the computing device (100) dynamically adjusts the image segmentation size (tile size, patch size) according to the object density, thereby preventing unnecessary information loss by segmenting into a small size in an area where objects are dense and analyzing them in detail, and reducing the number of operations by segmenting into a large size in an area where objects are sparse, thereby reducing the overall amount of operations, thereby maximizing resource efficiency through a balance between accuracy and speed, and improving processing speed. Meanwhile, a specific description of the process of determining the image segmentation size according to the object in the image by the computing device (100) will be described later with reference to FIG. 6.

According to one embodiment of the present disclosure, the computing device (100) can acquire segmented data per object for the acquired image (S140). For example, the computing device (100) can evaluate the segmented data per object based on preset evaluation criteria, and if the evaluation result does not pass the preset evaluation criteria, the computing device (100) can supplement the segmented data per object by utilizing the neural network model. Specifically, the computing device (100) can supplement the segmented data per object by utilizing the neural network model based on the context information of the acquired image. More specifically, if the reliability of the segmented data per object is lower than a preset threshold (e.g., 0.95) or if there is data that violates the drawing rules (e.g., mismatch in part number format), the computing device (100) can correct typos, incomplete text, misinterpreted abbreviations, etc. based on the context of the entire drawing (title column information, parts list, etc.), to supplement the segmented data per object. At this time, the computing device (100) can perform standardization on the acquired image in a format (Dictionary) of an object unit to obtain segmented data in an object unit. For example, the computing device (100) can perform standardization on the acquired image in a dictionary (JSON) format of a semantic unit object (Object) rather than a pixel. Specifically, the computing device (100) can read the acquired image with a parser, convert it into an intermediate representation (IR), generate JSON that conforms to a schema, unify the coordinate system and units, and then generate an ID that includes a type, a layer, semantic objects (semantics), etc. to perform standardization in a dictionary (JSON) format of a semantic unit object (Object). Through this, the computing device (100) can perform normalization, stable ID, and cost-minimizing matching on the acquired image to quickly identify a desired image through vector search, etc. even if there is no separate metadata such as tags, and utilize preprocessed data. Meanwhile, a detailed description of the process in which the computing device (100) supplements the segmented data of each object by utilizing the neural network model based on the context information of the image is described later with reference to FIG. 7.

FIG. 4 is a schematic diagram illustrating a process of acquiring an image including at least one object and identifying a technical field to which the acquired image corresponds according to one embodiment of the present disclosure.

Referring to FIG. 4, a computing device (100) may acquire an image (10) including at least one object. The acquired image (10) may include various drawings in technical fields, and for example, may include architectural/civil engineering CAD drawings, facility layout drawings, P&ID (Process & Instrumentation Diagram), electrical/PCB circuit diagrams, mold/machine processing drawings, drawings for patent applications, scanned paper-based drawings (PDF/image), etc., but is not limited to the above examples and drawings in various technical fields may be applied.

In addition, the image (10) including at least one object may include at least one of a title box, an annotation box, a part, or a drawing area. At this time, the title box may refer to metadata for tracking the identity, management, and version of the drawing, including the drawing number, drawing title, author, reviewer, approver name and signature, creation date, revision history, scale, drawing size (Size, A3, A4, etc.), company name/organization name, logo, etc. indicated at the lower right of the drawing (standardized by international standards ISO, KS, ANSI, etc.). In addition, the annotation box may refer to a text box in which a drawing interpreter writes a supplementary explanation so that the design intent is not misunderstood, such as a description of a specific part (e.g., “Dimension unit is mm”, “Tolerance ±0.1 mm”), compliance with laws/standards, references, or related drawing numbers. Additionally, the above-mentioned component may refer to an actual object expressed as a shape (line, circle, curve, etc.), and the above-mentioned drawing area may refer to the entire area where the drawing is expressed on paper (or screen), but is not limited thereto. Meanwhile, the computing device (100) can identify (11) the technical field to which the acquired image (10) corresponds, and a description thereof will be provided below.

According to one embodiment of the present disclosure, the computing device (100) can identify (11) the acquired image (10) as at least one of the fields of architectural technology, electrical technology, and mechanical technology. Specifically, if at least one of hatching/wall/slab/door/window symbols, scale bars, scales, room names (“Living, Bath, Corridor”), and plan/elevation/cross-section multi-views exists in the acquired image, the computing device (100) can identify (12) the acquired image as the fields of architectural technology and civil engineering, as in the example of FIG. 4. Alternatively, the computing device (100) can identify the acquired image (10) as the field of electrical technology, if at least one of circuit symbols (resistor R, capacitor C, ground, diode, transistor, connector), net names (VCC, GND, +5V), or wiring graphs exists in the acquired image (10). As another example, the computing device (100) can identify the acquired image (10) as a mechanical technology field if at least one of a geometric tolerance/surface roughness symbol, a fastener (bolt, nut), cross-sectional hatching, and a dimension/tolerance (Ø, ±0.1, M6x1) is present in the acquired image (10), but is not limited to the above example and various examples can be utilized.

For example, in the case of a PCB drawing, where the fine wiring width occupies an important part of the content of the drawing, if the computing device (100) applies the same ROI (large area focused) as the architectural standard in the process of performing object-based segmentation on the image (10), detailed wiring changes may be missed, resulting in inaccurate results. Therefore, through one embodiment of the present disclosure, the computing device (100) can first identify (11) the technical field of the acquired image (10), thereby accurately distinguishing a part with technical significance by reflecting the characteristics of each technical field, and utilize it in the process of preprocessing the drawing. Meanwhile, the computing device (100) can perform object-based segmentation on the acquired image based on the identified technical field by utilizing a neural network model, and a detailed description thereof will be described later with reference to FIG. 5.

FIG. 5 is a schematic diagram illustrating a process of performing object-level segmentation on the acquired image based on the identified technical field by utilizing a neural network model according to one embodiment of the present disclosure.

Referring to FIG. 5, the computing device (100) can perform segmentation (22) of the acquired image (10) on an object basis based on the identified technical field (12) by utilizing a neural network model (20). At this time, a pre-learned Transformer-based model, such as a large-scale language model (LLM) such as Google's Gemini or OpenAI's Chat-GPT, can be used as the neural network model (20), and a pre-learned model based on the relationship between images and texts for performing a multi-modal search can be included. As a specific example, the neural network model (20) may include a CLIP (Cooperative Learning of Image and Text Representations) model, a BLIP (Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation) model, etc., which are pre-trained through ImageNet, which includes a large amount of image-text pair learning data. However, in addition to the examples above, various neural network models that learn and connect the embedding of images and texts through the interaction between images and texts may be included in the neural network model (20).

For example, the computing device (100) may perform (22) segmentation of the acquired image (10) on the basis of the identified technical field (12) by utilizing the neural network model (20) based on preset criteria (21). At this time, the preset criteria (21) may include at least one of a title section segmentation criterion, a dimension line and annotation separation criterion, or an internal region of interest (ROI) setting criterion. Specifically, the computing device (100) may set the title section segmentation criterion to perform segmentation of the object within a certain ratio (e.g., bottom 15%, right 30%) compared to the sheet size (A0, A1, A2, A3, etc.) for the title section located at the bottom right of the drawing in accordance with most international standards (ISO, KS, ANSI, etc.). Alternatively, the computing device (100) may set criteria for separating dimension lines and annotations, such as “classifying geometric patterns (lines + arrows + text) as dimension lines, and classifying single text, text within blocks, and table-type text as annotations” to distinguish between object changes and annotation changes by separating the design object (part/pipe/structure) and dimension/description text. As another example, the computing device (100) may obtain criteria for setting an internal region of interest (ROI), such as an example of “selecting an area where objects such as lines, circles, symbols, and text are gathered at a certain density (0.9) or higher as an ROI” to include “an area where design objects are densely packed” for the internal region of interest (ROI) and exclude “ancillary information such as title columns and annotations.” However, in addition to the above examples, various examples may be utilized for the preset criteria (21).

Alternatively, the computing device (100) may utilize the neural network model (20) to set (21) a criterion for a region of interest (ROI) according to the identified technical field (12), and perform (22) segmentation of the acquired image into object units based on the criterion (21) for the set region of interest (ROI). For example, the computing device (100) may automatically divide a room-unit area into ROIs in the architecture/civil engineering field, where structures, safety-related matters, and spatial layout are important, and may preferentially set an ROI area for an object that meets building code standards, such as “stair step height, entrance width,” where compliance with regulations is important, and may set a criterion for a region of interest (ROI) by reinforcing an ROI area around a load-bearing member (column, beam) with high structural importance more than the average of the entire drawing. Alternatively, the computing device (100) may, in relation to the electrical/electronic/PCB field, expand the ROI centered on a specific net connection of a circuit node, and set a cluster with a higher component/wiring density than the average of the entire drawing as the ROI. However, in addition to the above example, various examples may be utilized in the process of setting the criteria for the focused area (ROI) according to the identified technical field (12). Through this, the computing device (100) reduces unnecessary operations by performing object-level segmentation on the acquired image through the criteria for the focused area (ROI) set according to the key object of each field and the purpose of review, and can preprocess the image by accurately identifying a technically meaningful area in an actual technical field. For example, as shown in FIG. 5, the computing device (100) sets (21) a standard for a region of interest (ROI) according to the architectural and civil engineering fields (12) for the acquired image (10), and performs segmentation (22) of the acquired image on an object basis based on the standard (21) for the set region of interest (ROI), thereby obtaining meaningful information about the image, such as “Building 101 is 84 square meters, Building 102 is 59 square meters.” Meanwhile, the computing device (100) can determine the segmentation size of the image according to the object in the image, and a specific description related thereto will be described later with reference to FIG. 6.

FIG. 6 is a schematic diagram illustrating a process of determining a segmentation size of an image according to an object in an acquired image and performing object-level segmentation on the acquired image based on the determined segmentation size, using a neural network model according to one embodiment of the present disclosure.

Referring to FIG. 6, the computing device (100) can determine (31 or 32) a segmentation size of the image according to an object in the acquired image (10), and perform object-by-object segmentation on the acquired image (10) based on the determined segmentation size (31 or 32). For example, the computing device (100) can determine the segmentation size of the image based on the density of objects in the acquired image (10) by utilizing the neural network model (20). More specifically, the computing device (100) can determine the segmentation size of the image by “determining the segmentation size of the image to be smaller than the average with a high resolution” (31) when the density of objects in the acquired image is higher than the average of the entire image, such as the left square, or by “determining the segmentation size of the image to be larger than the average with a low resolution” (32) when the density of objects in the acquired image is lower than the average of the entire image, such as the right square. Through this, the computing device (100) dynamically adjusts (31 or 32) the image segmentation size (tile size, patch size) according to the object density, thereby preventing unnecessary information loss by segmenting into a small size in an area where objects are dense and analyzing them in detail, and by segmenting into a large size in an area where objects are sparse, thereby reducing the number of operations and reducing the overall amount of operations, thereby maximizing resource efficiency and improving processing speed through a balance between accuracy and speed.

FIG. 7 is a schematic diagram illustrating a process of supplementing the segmented data of the object unit by utilizing the neural network model based on context information of the image according to one embodiment of the present disclosure.

Referring to FIG. 7, the computing device (100) performs an evaluation on the segmented data (23) of the object unit based on preset evaluation criteria, and if the evaluation result does not pass the preset evaluation criteria, the computing device (100) can supplement (23') the segmented data of the object unit by utilizing the neural network model. Specifically, the computing device (100) can supplement (23') the segmented data of the object unit by utilizing the neural network model (20) based on the context information of the acquired image. More specifically, the computing device (100) can supplement (23') the segmented data of the object unit by utilizing the neural network model (20). In other words, the computing device (100) can be used to evaluate the segmented data of the object unit if the reliability is lower than a preset threshold (e.g., 0.95), or if the area is not different according to the number of apartment buildings, such as "Building 101 is 84 square meters, Building 102 is 59 square meters." If data exists, typos, incomplete text, misinterpreted abbreviations, etc. can be corrected based on the context of the entire drawing (title information, parts list, etc.) to supplement (23') the segmented data in units of objects, such as "Building 101 is 59 square meters, Building 102 is 59 square meters." At this time, the computing device (100) can perform standardization on the acquired image (10) in a format (Dictionary) in units of objects to obtain segmented data (23) in units of objects. For example, the computing device (100) can perform standardization on the acquired image (10) in a dictionary (JSON) format of a semantic unit object (Object) rather than a pixel. Specifically, the computing device (100) reads the acquired image (10) with a parser, converts it into an intermediate representation (IR), generates JSON that conforms to the schema, unifies the coordinate system and units, and then generates an ID that includes a type, a layer, semantic objects, etc., to perform standardization in the dictionary (JSON) format of a semantic unit object (Object). Through this, the computing device (100) performs normalization, stable ID, and cost minimum matching on the acquired image (10), so that even if there is no separate metadata such as a tag, the desired image can be quickly identified through vector search, etc., and the preprocessed data can be utilized.

FIG. 8 is a simplified, general schematic diagram of an exemplary computing environment in which embodiments of the present disclosure may be implemented.

Although the present disclosure has been described above as being generally implemented by a computing device, those skilled in the art will appreciate that the present disclosure may also be implemented in combination with computer-executable instructions and/or other program modules that may be executed on one or more computers and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, data structures, and the like that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will appreciate that the methods of the present disclosure can be implemented with other computer system configurations, including single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like, each of which may be operatively connected to one or more associated devices.

The described embodiments of the present disclosure can also be practiced in distributed computing environments, where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Computer-readable media can be any media that can be accessed by a computer, and includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media. By way of example, and not limitation, computer-readable media can include computer-readable storage media and computer-readable transmission media. Computer-readable storage media includes both volatile and nonvolatile media, transitory and non-transitory media, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be accessed by a computer and used to store the desired information.

Computer-readable transmission media typically includes any information delivery media that embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. The term modulated data signal means a signal that has one or more of its characteristics set or changed so as to encode information in the signal. By way of example, and not limitation, computer-readable transmission media includes wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared, or other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An exemplary environment (1100) implementing various aspects of the present disclosure is illustrated, including a computer (1102) comprising a processing unit (1104), system memory (1106), and a system bus (1108). The system bus (1108) connects system components, including but not limited to the system memory (1106), to the processing unit (1104). The processing unit (1104) may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be utilized as the processing unit (1104).

The system bus (1108) may be any of several types of bus structures that may be additionally interconnected to a memory bus, a peripheral bus, and a local bus using any of a variety of commercial bus architectures. The system memory (1106) includes read-only memory (ROM) (1110) and random access memory (RAM) (1112). A basic input/output system (BIOS) is stored in non-volatile memory (1110), such as ROM, EPROM, or EEPROM, and includes basic routines that help transfer information between components within the computer (1102), such as during start-up. The RAM (1112) may also include high-speed RAM, such as static RAM, for caching data.

The computer (1102) also includes an internal hard disk drive (HDD) (1114) (e.g., EIDE, SATA) - which may also be configured for external use within a suitable chassis (not shown), a magnetic floppy disk drive (FDD) (1116) (e.g., for reading from or writing to a removable diskette (1118)), and an optical disk drive (1120) (e.g., for reading from or writing to a CD-ROM disk (1122) or other high-capacity optical media such as a DVD). The hard disk drive (1114), the magnetic disk drive (1116), and the optical disk drive (1120) may be connected to the system bus (1108) by a hard disk drive interface (1124), a magnetic disk drive interface (1126), and an optical drive interface (1128), respectively. The interface (1124) for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. In the case of the computer (1102), the drives and media correspond to storing any data in a suitable digital format. While the description of computer-readable media above refers to HDDs, removable magnetic disks, and removable optical media such as CDs or DVDs, those of ordinary skill in the art will appreciate that other types of computer-readable media, such as zip drives, magnetic cassettes, flash memory cards, cartridges, and the like, may also be used in the exemplary operating environment, and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules, including an operating system (1130), one or more application programs (1132), other program modules (1134), and program data (1136), may be stored in the drive and RAM (1112). All or portions of the operating system, applications, modules, and/or data may also be cached in RAM (1112). It will be appreciated that the present disclosure may be implemented in various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into the computer (1102) via one or more wired/wireless input devices, such as a keyboard (1138) and a pointing device such as a mouse (1140). Other input devices (not shown) may include a microphone, an IR remote control, a joystick, a game pad, a stylus pen, a touch screen, and the like. These and other input devices are often connected to the processing unit (1104) via an input device interface (1142) that is connected to the system bus (1108), but may be connected by other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, and the like.

A monitor (1144) or other type of display device is also connected to the system bus (1108) via an interface, such as a video adapter (1146). In addition to the monitor (1144), the computer typically includes other peripheral output devices (not shown), such as speakers, a printer, and so on.

The computer (1102) may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) (1148), via wired and/or wireless communications. The remote computer(s) (1148) may be a workstation, a computing device computer, a router, a personal computer, a portable computer, a microprocessor-based entertainment device, a peer device, or other conventional network node, and generally include many or all of the components described for the computer (1102), although for simplicity, only the memory storage device (1150) is shown. The logical connections shown include wired/wireless connections to a local area network (LAN) (1152) and/or a larger network, such as a wide area network (WAN) (1154). Such LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which may be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, the computer (1102) is connected to a local network (1152) via a wired and/or wireless communication network interface or adapter (1156). The adapter (1156) may facilitate wired or wireless communications to the LAN (1152), which may also include a wireless access point installed therein for communicating with the wireless adapter (1156). When used in a WAN networking environment, the computer (1102) may include a modem (1158), be connected to a communications computing device on the WAN (1154), or have other means of establishing communications over the WAN (1154), such as via the Internet. The modem (1158), which may be internal or external and wired or wireless, is connected to the system bus (1108) via a serial port interface (1142). In a networked environment, program modules or portions thereof described for the computer (1102) may be stored in a remote memory/storage device (1150). It will be appreciated that the network connections depicted are exemplary and other means of establishing a communications link between the computers may be used.

The computer (1102) operates to communicate with any wireless device or object that is arranged and operates via wireless communication, such as a printer, a scanner, a desktop and/or portable computer, a portable data assistant (PDA), a communication satellite, any equipment or location associated with a radio-detectable tag, and a telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, the communication may be a predefined structure as in a conventional network, or may simply be an ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) enables connections to the Internet and other devices without wires. Wi-Fi is a wireless technology that allows devices, such as computers, to send and receive data anywhere within the coverage area of a base station, both indoors and outdoors, similar to cell phones. Wi-Fi networks use wireless technologies called IEEE 802.11 (a, b, g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, to the Internet, and to wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz radio bands, at data rates of, for example, 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual-band).

Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

Those skilled in the art will appreciate that the various illustrative logical blocks, modules, processors, means, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, various forms of programs or design code (referred to herein, for convenience, as software), or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

The various embodiments presented herein can be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term article of manufacture includes a computer program, carrier, or media accessible from any computer-readable storage device. For example, computer-readable storage media include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory devices (e.g., EEPROMs, cards, sticks, key drives, etc.). Furthermore, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It should be understood that the specific order or hierarchy of steps in the presented processes is merely an example of exemplary approaches. It should be understood that the specific order or hierarchy of steps in the processes may be rearranged within the scope of the present disclosure based on design priorities. The appended method claims provide elements of various steps in a sample order, but are not intended to be limited to the specific order or hierarchy presented.

The description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments disclosed herein, but is to be construed in the broadest scope consistent with the principles and novel features disclosed herein.

Claims (24)

A method for processing an image by utilizing a neural network model, the method being performed by one or more processors of a computing device,
A step of obtaining an image containing at least one object;
A step of identifying the technical field to which the acquired image corresponds;
A step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing a neural network model; and
A step of obtaining segmented data of object units for the above-mentioned acquired image;
Including,
The step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the above neural network model is as follows:
A step of determining the segmentation size of the image according to the object in the acquired image by utilizing the above neural network model; and
A step of performing object-level segmentation on the acquired image based on the determined segmentation size is included.
The step of determining the image segmentation size according to the object in the acquired image by utilizing the above neural network model is as follows:
A step of determining the segmentation size of the image to be smaller than the average when the density of objects in the acquired image is higher than the average of the entire image; or
A step of determining the segmentation size of the image to be larger than the average when the density of objects in the acquired image is lower than the average of the entire image;
Containing at least one of,
method.
In the first paragraph,
The step of obtaining an image including at least one object comprises:
A step of obtaining an image including at least one of a title field, a comment box, a part, or a drawing area,
method.
In the second paragraph,
The step of identifying the technical field to which the above-mentioned acquired image corresponds is as follows:
A step of identifying the acquired image as at least one of the fields of architectural technology, electrical technology, and mechanical technology,
method.
In the first paragraph,
The step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the above neural network model is as follows:
A step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on preset criteria,
method.
In paragraph 4,
The step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on the preset criteria is as follows:
Criteria for dividing the title column;
Dimension line and annotation separation criteria; or
Criteria for setting internal region of interest (ROI);
A step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on at least one of the
method.
In the first paragraph,
The step of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the above neural network model is as follows:
A step of using the above neural network model to set criteria for a region of interest (ROI) according to the identified technology field; and
A step of performing object-level segmentation on the acquired image based on criteria for the set region of interest (ROI),
method.
delete delete delete In the first paragraph,
The step of obtaining segmented data for each object for the above acquired image is as follows:
A step of performing an evaluation on the segmented data of the object unit based on preset evaluation criteria; and
If the evaluation result does not pass the preset evaluation criteria, the method further includes a step of supplementing the segmented data of the object unit by utilizing the neural network model.
method.
In paragraph 10,
If the above evaluation result does not pass the preset evaluation criteria, the step of supplementing the segmented data of the object unit by utilizing the neural network model is as follows:
Comprising a step of supplementing the segmented data of the object unit by utilizing the neural network model based on the context information of the acquired image.
method.
In the first paragraph,
The step of obtaining segmented data for each object for the above acquired image is as follows:
A step of obtaining segmented data in units of objects by performing standardization on the acquired image in a format (Dictionary) in units of objects,
method.
A computer program stored in a computer-readable storage medium, wherein when the computer program is executed by one or more processors, the computer program causes the one or more processors to perform operations for processing an image using a neural network model, the operations comprising:
The act of obtaining an image containing at least one object;
An action of identifying the technical field to which the above-mentioned acquired image corresponds;
An operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing a neural network model; and
An operation of obtaining segmented data of an object unit for the above-mentioned acquired image;
Including,
An operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the above neural network model is as follows:
An operation of determining the segmentation size of an image according to an object in the acquired image by utilizing the above neural network model; and
An operation of performing object-level segmentation on the acquired image based on the determined segmentation size is included.
The operation of determining the segmentation size of the image according to the object in the acquired image by utilizing the above neural network model is as follows:
An operation of determining the segmentation size of the image to be smaller than the average when the density of objects in the acquired image is higher than the average of the entire image; or
An operation of determining a segmentation size of the image to be larger than the average when the density of objects in the acquired image is lower than the average of the entire image;
Containing at least one of,
A computer program stored on a computer-readable storage medium.
In paragraph 13,
The operation of obtaining an image containing at least one object comprises:
A title field, comprising an action of obtaining an image containing at least one of a comment box, a part, or a drawing area;
A computer program stored on a computer-readable storage medium.
In paragraph 14,
The operation of identifying the technical field to which the above acquired image corresponds is as follows:
Including an action of identifying the above-mentioned acquired image as at least one of the fields of architectural technology, electrical technology, and mechanical technology.
A computer program stored on a computer-readable storage medium.
In paragraph 13,
An operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the above neural network model is as follows:
An operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on preset criteria,
A computer program stored on a computer-readable storage medium.
In paragraph 16,
An operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on the preset criteria is as follows:
Criteria for dividing the title column;
Dimension line and annotation separation criteria; or
Criteria for setting internal region of interest (ROI);
An operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the neural network model based on at least one of the above,
A computer program stored on a computer-readable storage medium.
In paragraph 13,
An operation of performing object-level segmentation on the acquired image based on the identified technical field by utilizing the above neural network model is as follows:
An operation of setting a criterion for a region of interest (ROI) based on the identified technology field by utilizing the above neural network model; and
An operation of performing object-level segmentation on the acquired image based on criteria for the above-described set region of interest (ROI),
A computer program stored on a computer-readable storage medium.
delete delete delete In paragraph 13,
The operation of obtaining segmented data for each object for the above acquired image is as follows:
An operation of performing an evaluation on the segmented data of the object unit based on preset evaluation criteria; and
If the evaluation result does not pass the preset evaluation criteria, the operation of supplementing the segmented data of the object unit by utilizing the neural network model is further included.
A computer program stored on a computer-readable storage medium.
In paragraph 22,
If the above evaluation result does not pass the preset evaluation criteria, the operation of supplementing the segmented data of the object unit by utilizing the neural network model is as follows:
An operation that includes supplementing the segmented data of the object unit by utilizing the neural network model based on the context information of the acquired image.
A computer program stored on a computer-readable storage medium.
As a computing device,
Contains at least one processor,
At least one processor,
Obtain an image containing at least one object;
Identifying the technical field to which the above acquired image corresponds;
By utilizing a neural network model, segmentation of the acquired image into objects is performed based on the identified technical field; and
It is configured to obtain segmented data of object units for the above-mentioned acquired image,
By utilizing the above neural network model, segmentation of the acquired image into objects based on the identified technical field is performed.
By utilizing the above neural network model, the image segmentation size is determined according to the object in the acquired image; and
It includes performing object-unit segmentation on the acquired image based on the determined segmentation size,
By utilizing the above neural network model, the image segmentation size is determined according to the object in the acquired image.
If the density of objects in the acquired image is higher than the average of the entire image, determining the segmentation size of the image to be smaller than the average; or
When the density of objects in the acquired image is lower than the average of the entire image, determining the segmentation size of the image to be larger than the average;
Containing at least one of,
Computing device.