WO2023242540A1 - Procédés et systèmes de transformation et de récupération d'informations à partir de données de document à l'aide d'un apprentissage automatique - Google Patents

Procédés et systèmes de transformation et de récupération d'informations à partir de données de document à l'aide d'un apprentissage automatique Download PDF

Info

Publication number
WO2023242540A1
WO2023242540A1 PCT/GB2023/051519 GB2023051519W WO2023242540A1 WO 2023242540 A1 WO2023242540 A1 WO 2023242540A1 GB 2023051519 W GB2023051519 W GB 2023051519W WO 2023242540 A1 WO2023242540 A1 WO 2023242540A1
Authority
WO
WIPO (PCT)
Prior art keywords
document
user
query
question
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/GB2023/051519
Other languages
English (en)
Inventor
Gregorio Benedetto BENINCASA
Momchil Preslavov KONSTANTINOV
Josh Karlin
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
ETL Realisations 2024 Ltd
Original Assignee
Eigen Technologies Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Eigen Technologies Ltd filed Critical Eigen Technologies Ltd
Publication of WO2023242540A1 publication Critical patent/WO2023242540A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/93—Document management systems
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/332—Query formulation
    • G06F16/3329—Natural language query formulation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/33—Querying
    • G06F16/338—Presentation of query results

Definitions

  • Systems and methods herein may allow users to search through one or more documents by querying insights or ask intelligent questions.
  • the provided systems and methods may utilize deep learning models to understand a user’s intent (e g., desired information) without being intrusive and/or transform the document data into insights and retrieving the relevant insights taking into account the user’s knowledge.
  • the deep learning models herein may continuously learn and improve based on user feedback data or user inputs collected during the insights retrieval process thereby improving the accuracy of the delivered information subject to the user’s needs and computational budget.
  • the system herein may allow a user to specify the desired or interested information with improved flexibility. For instance, a user may be allowed to provide an input query in various forms such as search terms, an intelligent question, positive or negative keywords or any combination of the above in order to specify the desired information.
  • a method for querying insight information from a document comprises: providing, via a user interface, a plurality of different input fields for receiving a query input for searching insight information in a document, where the plurality of input fields comprise a first input field for receiving one or more questions in natural language and a second input field for receiving one or more search terms; receiving the query input submitted via the user interface, where the query input comprises a question or a search term; identifying, using one or more trained models executed at the backend computing system, one or more sections of the document each comprising a chunk of texts relevant to the query input, and rendering the one or more sections within the user interface with the chunk of texts annotated with a visual indicator; receiving a user feedback data via the user interface indicating an acceptance or rejection of the chunk of texts; and improving at least one of the trained models using the user feedback data.
  • the one or more models are trained using machine learning algorithms.
  • Another aspect of the present disclosure provides a non-transitory computer readable medium comprising machine executable code that, upon execution by one or more computer processors, implements any of the methods above or elsewhere herein.
  • the method further comprises, generating, at the backend computing system, a query based at least in part on the question.
  • the chunk of texts comprise an answer to the question.
  • the answer is annotated with a first visual indicator and the remaining texts in the chunk of texts are annotated with a second visual indicator.
  • the method further comprises generating a unified query by tokenizing at least the question and the unified query is used to identify one or more relevant sections. For example, the answer is identified by processing the one or more relevant sections using transformer model, and the transformer model is fine-tuned using a small amount of training datasets.
  • the method further comprises generating a unified query by tokenizing the question or the search term.
  • the method further comprises computing a sparse index score based at least in part on a similarity between the tokenized query and the one or more sections of the document retrieved by a sparse index.
  • the method further comprises computing a dense index score based at least in part on a similarity between a representation of the query and a representation of each section of the document retrieved by a dense index.
  • a pseudo-relevance score is generated based at least in part on the sparse index score and the dense index score and the method may further comprise generating a combined score by aggregating the sparse index score, the dense index score and the pseudo-relevance score using one of the one or more trained models.
  • the method further comprises retraining the one of the one or more trained models when the user feedback data comprises a number of relevant sections exceeding a predetermined threshold.
  • the plurality of different input fields further comprises a third input field for receiving a positive keyword and a fourth input field for receiving a negative keyword.
  • the query input further comprises a positive keyword or a negative keyword.
  • the positive keyword or the negative keyword is utilized to determine a relevance score for at least one of the one or more sections.
  • the user interface comprises a first panel displaying a set of sections ranked based at least in part on the combined score, and a second panel rendering at least one of the set of sections with the chunk of texts annotated with a visual indicator, and wherein the second panel displays one or more graphical elements for receiving the user feedback data.
  • the user interface is a web-based user interface.
  • the document comprises unstructured data.
  • the document is processed by the backend computing system for building a document dataset and indexes.
  • the method further comprises processing the one or more sections of the document and the query input by a large language model to output a generated answer.
  • Another aspect of the present disclosure provides a system comprising one or more computer processors and computer memory coupled thereto.
  • the computer memory comprises machine executable code that, upon execution by the one or more computer processors, implements any of the methods above or elsewhere herein.
  • FIG. 1 schematically shows a platform in which the method and system herein can be implemented.
  • FIG. 2 schematically illustrates a system for document analysis and information retrieval, in accordance with some embodiments of the present disclosure.
  • FIG. 3 and FIG. 4 shows various components and operations of an information extraction and retrieval system, in accordance with some embodiments of the present disclosure.
  • FIGs. 5-10 show examples of user interface (UI) provided by a frontend component of the system for insight query, in accordance with some embodiments of the present disclosure.
  • UI user interface
  • the present disclosure provides methods, systems and platforms for document processing, information and insight extraction and retrieval.
  • Systems and methods herein may beneficially augment a user’s analysis and understanding of document content by incorporating a variety of machine learning (ML) techniques such as machine learning (ML) for augmented intelligence, and other techniques such as heuristics injection and knowledge collection.
  • ML machine learning
  • the platform may advantageously maximize and/or optimize the use of a user’s knowledge while minizine the computational budget by improving the interaction between Human and Machine, incorporating Human Computer Interaction techniques, machine learning techniques (e.g., supervised, unsupervised, semi-supervised, trial design), knowledge base construction techniques and the like
  • methods and systems herein may allow for fast semantic search by computing one or more similarity scores generated with lightweight models.
  • the systems and platform herein may be capable of efficiently and effectively retrieving and extracting information from a universe of documents (e.g., raw documents).
  • the raw documents may comprise unstructured or semi- structured electronic document text.
  • TJnstructured text documents may contain “free text” in which the underlying information is mainly captured in the words themselves.
  • the unstructured document texts may include, for example, open text, images, that have no predetermined organization or design.
  • Semi-structured text may capture a significant portion of the information in the position, layout and format of the text but the information within has no structure.
  • the systems herein may be capable of extracting information and retrieving insights from the raw documents by converting the raw document texts into structured data (e.g., document datasets, indexes) then retrieving insight needed or desired by a user with machine learning techniques.
  • Methods and systems herein may improve the performance of insight retrieval by dynamically constructing the machine learning algorithm trained models using user feedback data.
  • the system and platform herein may train models to retrieve and extract information from a set of documents and a set of unseen documents by dynamically constructing models for retrieval and extraction, employ models benchmarks and model competition during training, improve the model training process and inform users when a model is properly trained, and/or use weak supervision for training large size models.
  • the models provided by the systems herein may adapt to a user’s need by continuously learning from user’s feedback or user’s interaction with the system during the insight retrieval process.
  • the user feedback data collected by the system may comprise clickthrough data (e g., how quickly a user responds to a system suggested answer, how many passages/answers identified by the system as relevant are confirmed (e.g., clicked on) or ignored by the user, etc.), or comprise user input indicative of the relevant information to the user in a given document (e g., system identified relevant information which the user may be interested in and the user may accept or reject the system identified relevant information or answer).
  • the user feedback collected by the system can comprise various other data such as whether the user used a system suggested search term and/or question for inputting a query.
  • the provided systems and methods may collect extracted knowledge or information for further improving insight querying.
  • the extracted knowledge or information may be managed and maintained in knowledge bases (e.g., object model with classes, sub-classes, instances or other structures for storing structured and unstructured information).
  • the systems and methods may employ various other suitable document processing techniques such as summarization, document diffing, coreference resolution and relation extraction, template filing, normalization of extracted fields and the like.
  • the provided methods and systems can be implemented in various scenarios such as in cloud or an on-premises environment.
  • ком ⁇ онент can be a processor, a process running on a processor, an object, an executable, a program, a storage device, and/or a computer.
  • an application running on a server and the server can be a component.
  • One or more components can reside within a process, and a component can be localized on one computer and/or distributed between two or more computers.
  • these components can execute from various computer readable media having various data structures stored thereon.
  • the components can communicate via local and/or remote processes such as in accordance with a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or across a network, e.g., the Internet, a local area network, a wide area network, etc. with other systems via the signal).
  • a signal having one or more data packets (e.g., data from one component interacting with another component in a local system, distributed system, and/or across a network, e.g., the Internet, a local area network, a wide area network, etc. with other systems via the signal).
  • a component can be an apparatus with specific functionality provided by mechanical parts operated by electric or electronic circuitry; the electric or electronic circuitry can be operated by a software application or a firmware application executed by one or more processors; the one or more processors can be internal or external to the apparatus and can execute at least a part of the software or firmware application.
  • a component can be an apparatus that provides specific functionality through electronic components without mechanical parts; the electronic components can include one or more processors therein to execute software and/or firmware that confer(s), at least in part, the functionality of the electronic components.
  • a component can emulate an electronic component via a virtual machine, e.g., within a cloud computing system.
  • methods and systems herein may provide various functions that can be implemented or accessed via web application program interfaces (APIs), a Software Development Kit (SDK), web-based integrated development environment (IDE) and the like.
  • APIs web application program interfaces
  • SDK Software Development Kit
  • IDE web-based integrated development environment
  • Various components of the system herein may be seamlessly integrated into a third-party platform or system via customized Software Development Kit (SDK) or APIs.
  • SDK Software Development Kit
  • intelligent information extraction and retrieval as well as document processing modules may be provided via open-ended integration with a full suite of APIs and plugins thereby allowing for convenient and seamless system integrations into any third-party systems.
  • the platform may be a no-code user-friendly platform that requires only a small amount of training datasets to deliver highly accurate results across a wide range of document types and data formats.
  • the platform herein may train models to be able to extract relevant information relating to a user input from documents.
  • the data extraction models may be trained on a relatively small number of documents, and still provide accurate outcomes when used to analyze documents.
  • the platform may provide customized models using limited training datasets (e g., 2-50 examples) to fine tune the models to extract and retrieve information from any document, for any data, any user and any use case. This greatly reduces the time and effort required by a user before the system can commence useful data extraction.
  • the present disclosure also provides a user-friendly user interface (UI) which allows user to specify the information needed in natural language-based question, using search terms, positive or negative keywords of any combination of the above with improved flexibility.
  • UI user-friendly user interface
  • the UI may also allow the user to interact with the extracted information and collect user feedback data related to the relevancy of the extracted information which user feedback data may be utilized by the system to further improve the information retrieval models in an automated fashion.
  • the system herein may provide improved flexibility for a user to provide input via a user interface to specify the desired or interested information or to retrieve information from one or more documents.
  • the system herein may allow a user to provide an insight query input in various formats or types. For instance, a user may be allowed to provide search terms, an intelligent question, positive or negative keywords or any combination of the above in order to specify the desired information.
  • the system may process the user input in the one or more types or input channels in respective processes and identify the relevant information by aggregating a plurality of similarity and relevancy scores as well as the real-time user feedback data in a unique process.
  • the output provided by the system may comprise the relevant information and/or answer in response to the user input querying the information.
  • relevant information as utilized herein may generally refer to target of information to satisfy the user’s information needs or texts in one or more sections from the original document (e.g., highlight salient passages) relevant to the user’s query such that the user can make informed decisions based on such information,
  • the system output may comprise relevant pieces of information which the user may rely on to determine/satisfy some criterion.
  • the term “answer” as utilized herein may refer to a word, short phrase or a span of texts that directly answer a question specified in the query input. In some cases, an “answer” may not be provided if there is no question specified in the query input, when the question cannot be answered by extractive texts from the original document, and/or when the information collected by the retriever is insufficient to answer the provided question.
  • the system may employ different back-end processes or models to address the different types/formats of queries (e g., intelligent question, search terms, positive or negative keywords, etc.) to enable accurate extraction of “answer” (words/short phrases) and relevant “sections” (e.g., paragraphs/clauses) from diverse text, enabling documents to be turned into actionable data.
  • words and short phrases may be considered "answer” and are extracted in a question-answer process or utilizing a question-answering model (reader engine). Longer chunks of text may be considered “sections”, and may be extracted in a section extraction process.
  • document data may also be extracted from semi-structured data such as tables in a table extraction process.
  • the various different models for extracting the answer and relevant passage/section may generally refer to data extraction models but with different input features or network architecture.
  • the document may be tokenized and may be processed to build document datasets.
  • indexes are generated for the datasets.
  • the indexes may comprise sparse index each may index or correspond to an element such as a token, a word, a passage or other data structures for efficient lookup. It should be noted that any other suitable data structures may be used for the lookup of the elements.
  • the indexes may also comprise a dense index each may correspond to or index elements such as a representation of textual items (e.g., word embeddings of a chunk of texts).
  • a word embedding (a real-valued vector that encodes the meaning of the word such that the words that are closer in the vector space are expected to be similar in meaning) may be indexed and a dense index may be created.
  • a chunk may, for example, be a sentence, a section, a passage, a paragraph, a clause, or a sub-clause of a document.
  • the chunks may be defined such that they correspond to the format of the document. For example, chunks may not carry over from one part of the document, such as a chapter or subchapter, to another or from one paragraph to another.
  • chunks of text may be terminated by a full stop. In some cases, at least some chunks may contain multiple words.
  • a representation of textual items may be indexed and a dense index may be created.
  • a dense index for a section may correspond to the representation or word embeddings of the section.
  • section as utilized herein may be sentence-level, or longer sections (e.g., paragraphs).
  • the sparse index and dense index may be built for both the raw document and input query for information retrieval. Alternatively, the sparse index and dense index may be built only for the document not the query. Details about building the sparse index and dense index are described later herein.
  • the term “question” as utilized herein, may generally refer to a question in natural language (e.g., “what is the governing law?”, “what is the operational risk event predicted for file x being delayed from source system A?”) unless the context suggests otherwise.
  • the system herein may provide an “answer” to a question or multiple most likely answers to a question.
  • the answer may be a piece of text in one or more sections or passages which answers the question.
  • the extracted answer may be presented at the user interface by rendering the corresponding characters (text) of the original document and automatically highlighting the corresponding characters in one or more sections of the original document.
  • the insight query results may also comprise annotated relevant information (e.g., highlight, underline or any visual indicator) that may or may not encompassing the answer.
  • an answer may comprise a string of text, generated by a generative Large Language Model (LLM).
  • LLM generative Large Language Model
  • an answer may be generated by a generative LLM based on one or more passages identified in response to a user’s question.
  • the answer may or may not be exactly the original texts that are processed by the LLM to form the answer.
  • the answer may be generated in natural language where the original texts may serve as supporting evidence.
  • the answer (generated answer) may be presented at the user interface by rendering the corresponding characters (text), preceded by the string “Generated:” and automatically highlighting the corresponding one or more sections of the original document as supporting evidence for the generated answer.
  • search term may generally refer to query terms unless the context suggests otherwise.
  • the one or more search terms may be used to search the relevant information such as based on intent match, phrase match, semantic matching, broad match or others.
  • the system may employ one or more algorithms or rules to interpret the query terms and provide results that match the search intent (e.g., results may not contain the exact term). For instance, latent semantic indexing, co-occurring terms and synonyms or other natural language processing techniques may be utilized to provide the search results.
  • the search results may comprise the search terms in any order, case insensitive, a word as a truncated search term (e.g., “governed”, “governing”, “govern” are identified with “govern”), or a term has a similar semantic meaning of the search term (e.g., search results containing “salary” or “compensation” may be returned for search term of “remuneration”).
  • a word as a truncated search term e.g., “governed”, “governing”, “govern” are identified with “govern”
  • search results containing “salary” or “compensation” may be returned for search term of “remuneration”.
  • the term “positive keywords” or “negative keywords” as utilized herein may be treated as soft constraints for determining relevancy of query results.
  • the positive keywords or negative keywords may not form part of the query.
  • the system may use the presence of a keyword in a section (if positive keyword is specified) to determine a relevancy of the section. Additionally, the system may use presence of a keyword in a section to determine the section as “non-relevant” if the keyword is specified as a negative keyword. In some cases, absence of a keyword in a section may also be used to determine whether the section is relevant or non-relevant. In some cases, one or more sections may be first identified as relevant based on the question and/or search terms then further refined based on the keywords.
  • one or more sections may be identified in response to a query constructed based on the question and/or search terms, and if a positive keyword is matched with a word in an identified section, the section may be labeled as relevant. Similarly, if a negative keyword is matched with a word in an identified section, the section may be labeled as irrelevant. In some cases, the match may not be exact literal match.
  • a match of the positive keywords or negative keywords may be determined using any suitable algorithms. For example, the match may be determined based on regex (regular expression) features where pattern matching may be utilized to search for particular strings of characters rather than constructing multiple, literal search queries.
  • FIG. 1 schematically shows a platform 100 in which the method and system herein can be implemented.
  • a platform 100 may include one or more user devices 101-1, 101-2, 101-3, a server 120, a system 121, one or more third-party systems 130, and a storage unit 111, 123.
  • Each of the components 101-1, 101-2, 101-3, 111, 120, 121, 123, 130 may be operatively connected to one another via a network 110 or any type of communication link that allows transmission of data from one component to another.
  • the system 121 may be configured to permit users to perform insight query or information search through one or more documents.
  • the system 121 may include a plurality of functional components such as retriever engine, reader engine, recommending system, user interface module, model creation and management system, and/or various others described elsewhere herein. Details about the various functional components are described later herein.
  • the system 121 may be configured to train and develop a plurality of predictive models (e.g., RNN, CNN, GAN, classifiers, etc.) consistent with the methods and functions described herein.
  • the system 121 may be configured to perform one or more operations and provide one or more features consistent with those disclosed with respect to FIGs. 2-10.
  • the system 121 may provide pre-trained models and may fine tune the pretrained models with custom or private datasets to provide customized models.
  • the system may initialize encoder-decoder model with pre-trained encoder and/or decoder checkpoints (e.g., Bidirectional Encoder Representations from Transformers (BERT), Generative Pre-trained Transformer 2 (GPT-2)) to skip the costly pre-training.
  • the system 121 may further comprise binary and multi -category classification models to optimize the ranking of the retrieval results generated by the system, to enrich the user experience and improve the accuracy of the retrieval performance based on the user’s intent/knowledge.
  • the generative models and/or the classification models may be transformer-based models which may be fine-tuned using custom datasets.
  • the datasets may be generated manually such as by manual labelling, collected by the system (e.g., user clickthrough data) or generated automatically or semi-automatically by a labeling system.
  • custom datasets may be utilized to fine-tune a preliminary model (e g., pretrained model).
  • insights extracted by the system or newly collected user feedback data may be used to retrain or update a predictive model. Details about the transformer model, artificial intelligence (Al) engines and the training methods are described later herein
  • the system may be implemented in a cloud-based platform.
  • the front-end of the system may be implemented as a web application using the framework (e.g., Django Python) hosted on an Elastic Cloud Compute (EC2) instance on Amazon Web Services (AWS).
  • the backend of the system may be implemented as serverless compute service such as hosted on AWS Lambda as a serverless compute service running a web framework for developing RESTful APIs (e.g., FastAPI).
  • RESTful APIs e.g., FastAPI
  • the backend system e.g., AWS Lambda
  • the system 121 may be implemented anywhere within the platform, and/or outside of the platform 100. In some embodiments, system 121 may be implemented on server 120. In other embodiments, a portion of the system 121 may be implemented on the user device. Additionally, a portion of system 121 may be implemented on the third-party system 130. Alternatively or in addition to, a portion of the system 121 may be implemented in one or more storage units (e.g., knowledge base, data lakes, databases) 111, 123. The system 121 may be implemented using software, hardware, or a combination of software and hardware in one or more of the above- mentioned components within the platform. Details about the plurality of the components of the system 121 are described later herein.
  • a user 103-1, 103-2 may be associated with one or more user devices 101-1, 101-2, 101-3.
  • User device 101-1, 101-2, 101-3 may be a computing device configured to perform one or more operations consistent with the disclosed embodiments.
  • Examples of user devices may include, but are not limited to, laptop or notebook computers, desktop computers, mobile devices, smartphones/cell phones, wearable device (e.g., smartwatches), tablets, personal digital assistants (PDAs), media content players, television sets, video gaming station/system, virtual reality systems, augmented reality systems, microphones, or any electronic device capable of analyzing, receiving (e g., receiving user input indicating accept, reject or select a system identified relevant information or answer, user input for conducting insight querying, user input for modifying ruleset, etc ), providing or displaying certain types of data (e.g., rendering of a GUI displaying query results, highlighting relevant information and/or answer, rendering document, etc.) to a user.
  • the user device may be portable. In some cases, the user device may be located remotely from a human user, and the user can control the user device using wireless and/or wired communications.
  • the user device can be any electronic device with a display.
  • User device 101-1, 101-2, 101-3 may include one or more processors that are capable of executing non-transitory computer readable media that may provide instructions for one or more operations consistent with the disclosed embodiments.
  • the user device may include one or more memory storage devices comprising non-transitory computer readable media including code, logic, or instructions for performing the one or more operations.
  • the user device may include software applications that allow the user to search or query information in one or more documents (e.g., software application provided by third-party server 130), and/or software applications provided by the system 121 that allow the user device to communicate with and transfer data between server 120, the system 121, and/or the storage unit (e.g., knowledge base or database 111).
  • the user device 101-1, 101-2, 101-3 may include a communication unit, which may permit the communications with one or more other components in the platform 100.
  • the communication unit may include a single communication module, or multiple communication modules.
  • the user device may be capable of interacting with one or more components in the platform 100 using a single communication link or multiple different types of communication links.
  • User devices 101-1, 101-2, 101-3 may include a display.
  • the display may be a screen.
  • the display may or may not be a touchscreen.
  • the display may be a light-emitting diode (LED) screen, OLED screen, liquid crystal display (LCD) screen, plasma screen, or any other type of screen.
  • the display may be configured to show a user interface (UI) or a graphical user interface (GUI) rendered through an application (e.g., via an application programming interface (API) executed on the user device).
  • the GUI may display, for example, a user portal with various features such as document upload, query input field, preview of system identified relevant information, relevant sections, extractive answer, and the like.
  • the user device may also be configured to display webpages and/or websites on the Internet.
  • One or more of the web pages/web sites may be hosted by server 120, the third-party system 130 and/or rendered by the system 121.
  • users may utilize the user devices to interact with the system 121 or the third-party system 130 by way of one or more software applications (i.e., client software) running on and/or accessed by the user devices, wherein the user devices and the system 121 or the third- party system 130 may form a client-server relationship.
  • client software i.e., client software
  • the user devices may run dedicated mobile applications or software applications for accessing the client portal provided by the system 121 or the third-party system 130.
  • the software applications for managing the platform e.g., admin portal
  • document processing e.g., and for conducting insight query may be different applications.
  • the client application may comprise different interfaces/modes for a user to modify/specify heuristics for determining relevancy, perform insight query and view query result, select, reject or accept system identified relevant information, sections or answers, to manage the Al engine or handcrafted rules, respectively.
  • the client software i.e., software applications installed on the user devices 101-1, 101-2, 101-3
  • the client software can be available either as downloadable software or mobile applications for various types of computer devices.
  • the client software can be implemented in a combination of one or more programming languages and markup languages for execution by various web browsers.
  • the client software can be executed in web browsers that support JavaScript and HTML rendering, such as Chrome, Mozilla Firefox, Internet Explorer, Safari, and any other compatible web browsers.
  • the various embodiments of client software applications may be compiled for various devices, across multiple platforms, and may be optimized for their respective native platforms.
  • the provided platform may generate one or more graphical user interfaces (GUIs).
  • GUIs may be rendered on a display screen on a user device 101-1, 101-2, 101-3.
  • a GUI is a type of interface that allows users to interact with electronic devices through graphical icons and visual indicators such as secondary notation, as opposed to text-based interfaces, typed command labels or text navigation.
  • the actions in a GUI are usually performed through direct manipulation of the graphical elements.
  • GUIs can be found in handheld devices such as MP3 players, portable media players, gaming devices and smaller household, office and industry equipment.
  • the GUIs may be provided in software, a software application, a mobile application, a web browser, or the like.
  • the GUIs may be displayed on a user device (e.g., desktop computers, laptops or notebook computers, mobile devices, smart phones, personal digital assistants (PDAs), and tablets).
  • PDAs personal digital assistants
  • User devices may be associated with one or more users.
  • a user may be associated with a unique user device.
  • a user may be associated with a plurality of user devices.
  • a user may be registered with the platform.
  • user profile data may be stored in a database (e.g., database 123) along with a user ID uniquely associated with the user.
  • the user profile data may include, for example, user names, user ID, identity, business field, contact information, historical data, and various others.
  • a registered user may be permitted to share or publish exported insight information with other users or store the insight information in a storage space provided by the system.
  • a server 120 may access and execute the system 121 to perform one or more processes consistent with the disclosed embodiments.
  • the system may be software stored in memory accessible by a server (e.g., in memory local to the server or remote memory accessible over a communication link, such as the network).
  • the system(s) may be implemented as one or more computers, as software stored on a memory device accessible by the server, or a combination thereof
  • one or more systems or components of the present disclosure are implemented as a containerized application (e.g., application container or service containers).
  • the application container provides tooling for applications and batch processing such as web servers with Python or Ruby, JVMs, or Hadoop or HPC tooling.
  • the various functions performed by the system such as document processing, retriever-reader pipelines, generating ruleset for further modifying Al predictions, model manage system for training a predictive model, executing a trained model, updating and retraining a model using user feedback data and the like may be implemented in software, hardware, firmware, embedded hardware, standalone hardware, application specific-hardware, or any combination of these.
  • the system, and techniques described herein may be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof.
  • These systems, devices, and techniques may include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, graphics processing unit (GPU), coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
  • a programmable processor which may be special or general purpose, graphics processing unit (GPU), coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.
  • GPU graphics processing unit
  • the third-party system 130 can be any existing platforms or systems that provide document processing, data management and the like.
  • the third-party system may provide software applications to process and analyze document and the insight retrieval functions may be integrated into the applications running on the third-party system 130.
  • the third-party system may be hosted by any entity, organization that is interested in the data generated by the system herein.
  • the third-party system may utilize the data for various purposes or via privacy -preserving techniques such as, but not limited to, federated learning.
  • the third-party system may be in direct communication with the system 121 such that the document processing, information retrieval and the like may be integrated into the third-party application such as via an API.
  • the server 120 may also be configured to store, retrieve, and/or analyze data and information stored in one or more of the storage unit (e.g., knowledge base, databases).
  • the data and information may include converted document dataset, indexes (e.g., sparse index, dense index, etc.), extracted information or insight (e.g., locations and offset of tokens identifying start and end of relevant information or answer), user feedback data, query input data, data about a predictive model (e.g., parameters, model architecture, training dataset, performance metrics, threshold, etc.), and the like.
  • FIG. 1 illustrates the server as a single server, in some embodiments, multiple devices may implement the functionality associated with a server.
  • a server may include a web server, an enterprise server, or any other type of computer server, and can be computer programmed to accept requests (e.g., HTTP, or other protocols that can initiate data transmission) from a computing device (e.g., user device) and to serve the computing device with requested data.
  • a server can be a broadcasting facility, such as free-to-air, cable, satellite, and other broadcasting facility, for distributing data.
  • a server may also be a server in a data network (e.g., a cloud computing network).
  • a server may include known computing components, such as one or more processors, one or more memory devices storing software instructions executed by the processor(s), and data.
  • a server can have one or more processors and at least one memory for storing program instructions.
  • the processor(s) can be a single or multiple microprocessors, field programmable gate arrays (FPGAs), or digital signal processors (DSPs) capable of executing particular sets of instructions.
  • Computer-readable instructions can be stored on a tangible non-transitory computer-readable medium, such as a hard disk, a CD-ROM (compact disk-read only memory), and MO (magnetooptical), a DVD-ROM (digital versatile disk-read only memory), a DVD RAM (digital versatile disk-random access memory), or a semiconductor memory.
  • a tangible non-transitory computer-readable medium such as a hard disk, a CD-ROM (compact disk-read only memory), and MO (magnetooptical), a DVD-ROM (digital versatile disk-read only memory), a DVD RAM (digital versatile disk-random access memory), or a semiconductor memory.
  • the methods can be implemented in hardware components or combinations of hardware and software such as, for example, ASICs, special purpose computers, or general purpose computers.
  • Network 110 may be a network that is configured to provide communication between the various components illustrated in FIG. 1.
  • the network may be implemented, in some embodiments, as one or more networks that connect devices and/or components in the network layout for allowing communication between them.
  • user device 101-1, 101-2, 101-3 third-party system 130, server 120, system 121, and storage units 111, 123 may be in operable communication with one another over network 110.
  • Direct communications may be provided between two or more of the above components.
  • the direct communications may occur without requiring any intermediary device or network.
  • Indirect communications may be provided between two or more of the above components.
  • the indirect communications may occur with aid of one or more intermediary devices or networks. For instance, indirect communications may utilize a telecommunications network.
  • Indirect communications may be performed with aid of one or more routers, communication towers, satellites, or any other intermediary device or network.
  • types of communications may include, but are not limited to: communications via the Internet, Local Area Networks (LANs), Wide Area Networks (WANs), Bluetooth, Near Field Communication (NFC) technologies, networks based on mobile data protocols such as General Packet Radio Services (GPRS), GSM, Enhanced Data GSM Environment (EDGE), 3G, 4G, 5G or Long Term Evolution (LTE) protocols, Infra-Red (IR) communication technologies, and/or Wi-Fi, and may be wireless, wired, or a combination thereof.
  • the network may be implemented using cell and/or pager networks, satellite, licensed radio, or a combination of licensed and unlicensed radio.
  • the network may be wireless, wired, or a combination thereof.
  • User device 101-1, 101-2, 101-3, third-party system 130, server 120, or system 121 may be connected or interconnected to one or more storage units (e.g., databases, knowledge bases) 111, 123.
  • the databases may be one or more memory devices configured to store structured data. Additionally, the databases may also, in some embodiments, be implemented as a computer system with a storage device.
  • the database may make it easy to access ORC, Parquet, Avro or any other types of data stored in object store.
  • the databases may be used by components of the network layout to perform one or more operations consistent with the disclosed embodiments.
  • One or more local databases, and cloud databases of the platform may utilize any suitable database techniques.
  • structured query language or “NoSQL” database may be utilized for storing the document data, indexes, data generated by a predictive model such as extracted insight (e.g., relevant section, information, answer, etc.), and the like.
  • Some of the databases may be implemented using various standard data-structures, such as an array, hash, (linked) list, struct, structured text file (e.g., XML), table, JavaScript Object Notation (JSON), NOSQL and/or the like.
  • Such data-structures may be stored in memory and/or in (structured) files.
  • an object-oriented database may be used.
  • Object databases can include a number of object collections that are grouped and/or linked together by common attributes; they may be related to other object collections by some common attributes. Object-oriented databases perform similarly to relational databases with the exception that objects are not just pieces of data but may have other types of functionality encapsulated within a given object.
  • the database may include a graph database that uses graph structures for semantic queries with nodes, edges and properties to represent and store data. Also, the database may be implemented as a mix of data structures, objects, and relational structures. Databases may be consolidated and/or distributed in variations through standard data processing techniques. Portions of databases, e.g., tables, may be exported and/or imported and thus decentralized and/or integrated.
  • the storage unit may comprise knowledge bases utilized to store complex structured and unstructured information generated and retrieved by the system.
  • the knowledge base may be an object model with classes, subclasses and instances for storing, for example, user feedback data, extracted information and various other data and information as described elsewhere herein.
  • the platform 100 may construct the database for fast and efficient data retrieval, query and delivery.
  • the system 121 may provide customized algorithms to extract, transform, and load (ETL) the data.
  • the system 121 may construct the databases using proprietary database architecture or data structures to provide an efficient database model that is adapted to large scale databases, is easily scalable, is efficient in query and data retrieval, or has reduced memory requirements in comparison to using other data structures.
  • the one or more storage systems 123, 111 which may be configured for storing or retrieving relevant data as described elsewhere herein.
  • the system 121 may source data or otherwise communicate (e.g., via the one or more networks 110) with one or more external systems or data sources 111 (e.g., document storage), and third party system 130.
  • the system 121 may retrieve data from the storage systems 111, 123 which are in communication with the one or more external systems (e.g., external document management system, etc.) or third-party systems 130 (e.g., industry or company proprietary systems, etc.).
  • the storage systems can store algorithms or ruleset utilized by one or more methods disclosed herein.
  • one or more of the databases may be colocated with the server, may be co-located with one another on the network, or may be located separately from other devices.
  • One of ordinary skills will recognize that the disclosed embodiments are not limited to the configuration and/or arrangement of the database(s).
  • data stored in the knowledge base, databases or external databases can be utilized or accessed by a variety of applications through application programming interfaces (APIs). Access to the database may be authorized at per API level, per data level (e g., type of data), per application level or according to other authorization policies.
  • APIs application programming interfaces
  • Various aspects of the present disclosure may be applied to any of the particular applications set forth below or for any other types of applications or systems.
  • Systems or methods of the present disclosure may be employed in a standalone manner, or as part of a package.
  • the system may also allow for an easy and flexible integration of the various personalization features into any existing third-party website or platforms.
  • the system may provide a plurality of options such as raw application programming interface (API), Plugins, SDK, Google Tag Manager and the like for integrating the Al-based outputs (e g., extracted information, relevant sections, answer to a question, etc.) to a third-party platform.
  • the system may create various API endpoints for rending frontend elements and code injection.
  • One or more features (e.g., insight query, document processing, etc.) of the system may be integrated to a third-party application (e.g., company’s proprietary software, document management system, etc.).
  • a third-party application e.g., company’s proprietary software, document management system, etc.
  • the system may include a family of plugins, extensions, modules and scripts that facilitate development and integration of the document analysis, insight retriever-reader and services into third-party platforms.
  • FIG. 2 schematically illustrates a system 200 for document processing and information retrieval, in accordance with embodiments of the present disclosure.
  • the system 200 can be the same as the system 121 as described in FIG. 1.
  • the system may be implemented on a cloud platform system (e.g., including a server or serverless) that is in communication with one or more user systems/devices via a network.
  • the cloud platform system may be configured to provide the aforementioned functionalities to the users via one or more user interface 210.
  • the user interface may comprise a graphical user interfaces (GUIs), which may include, without limitation, web-based GUIs, clientside GUIs, or any other GUI as described above.
  • GUIs graphical user interfaces
  • a user may upload document and perform insight query via a web-based-GUIs or within a web browser.
  • GUI graphical user interface
  • GUIs are types of interface that allows users to interact with electronic devices through graphical icons and visual indicators such as secondary notation, as opposed to text-based interfaces, typed command labels or text navigation.
  • the actions in a GUI are usually performed through direct manipulation of the graphical elements.
  • GUIs can be rendered in hand-held devices such as mobile devices, MP3 players, portable media players, gaming devices and smaller household, office and industry equipment.
  • the GUIs may be provided in a software, a software application, a web browser, etc.
  • the GUIs may be displayed on a user device or user system (e.g., mobile device, personal computers, personal digital assistants, cloud computing system, etc.).
  • the GUIs may be provided through a mobile application or web application.
  • GUI graphical user interface
  • the display may or may not be a touchscreen.
  • the display may be a light-emitting diode (LED) screen, organic light-emitting diode (OLED) screen, liquid crystal display (LCD) screen, plasma screen, or any other type of screen.
  • LED light-emitting diode
  • OLED organic light-emitting diode
  • LCD liquid crystal display
  • plasma screen or any other type of screen.
  • the system 200 may comprise a front-end component 220 allowing users to access the system.
  • the user interface (UI) 210 or a graphical user interface (GUI) may be rendered through an application (e.g., via an application programming interface (API) executed on the user device or the user system, or on the cloud) of the front-end component 220.
  • API application programming interface
  • the frontend component 220 may receive user input in one or more forms such as question, search term, positive keyword and/or negative keyword and present a search result such as a structured data output comprising relevant information with respect to the question, search terms and/or keywords, or “answers” to the question.
  • the user input can include any other query information such as specifying data types to be sorted (e.g., dates, amounts, etc.).
  • the frontend component 220 may also comprising a UI module for rendering one or more GUIs such as a GUI for uploading document(s), GUI for receiving the user query input in a plurality of input channels (e.g., question, search term, positive keyword and/or negative keyword), GUI for displaying the query result and GUI for receiving and collecting user feedback data. Examples of the user interface, user input fields and output of the system presented within a GUI are described later herein.
  • the frontend component 220 may allow a user to upload one or more documents (electronic document) to be analyzed by the system for information retrieval.
  • the UI may provide a landing page where the user can upload one or more documents into the platform to start a session for document processing and information retrieval.
  • the frontend component 220 may be in communication with a backend component 230.
  • the user-uploaded document may be processed by the document dataset builder and indexer 237 of the system 200 to build document dataset and build indexes.
  • the document dataset builder and indexer 237 is shown as a component separate from the backend component, one or more components of the document dataset builder and indexer 237 can be an integral part of the backend component.
  • the document dataset builder and indexer 237 may process an uploaded document to build a document dataset by extracting textual data, decomposing the uploaded document, and preprocessing the text.
  • a user-uploaded document may be converted, as necessary, from the format in which it is uploaded, for example PDF, to a raw text or html/DOCX representation etc. (in which HTML or other metadata tags may, in some cases, be used to represent document structure, formatting etc.).
  • original document as utilized herein may refer to the converted document, i.e. after such conversion has been applied (e.g. in raw text, HTML or DOCX form, as opposed to a pre-converted PDF).
  • the textual data may be extracted using any suitable algorithm. For example, if the uploaded document is PDF or image file, it may be converted to text data using OCR (Optical Character Recognition) techniques.
  • OCR Optical Character Recognition
  • the dataset building may further comprise document decomposition.
  • the document dataset builder and indexer 237 may utilize any suitable document analysis techniques to tokenize the document.
  • the document dataset builder and indexer 237 may section the document by merging sentences or utilization of heuristics over pixel values in the PDF to identify ‘chunks’ of text or more advanced document segmentation techniques that account for both the visual and the layout information in the document.
  • the document dataset builder and indexer 237 may merge the chunks to form sections.
  • a chunk may, for example, be a sentence, a paragraph, a clause, or a sub-clause of a document.
  • the chunks may be defined such that they correspond to the format of the document.
  • chunks may not carry over from one part of the document, such as a chapter or sub-chapter, to another or from one paragraph to another.
  • chunks of text may be terminated by a full stop.
  • chunks may contain multiple words.
  • a section formed by the merging of chunks may correspond to a paragraph, sentence, clause etc. depending on the precise form of tokenization applied.
  • the dataset building may also comprise text preprocessing.
  • the text preprocessing may comprise a variety of operations such as stopwords removal, stripping non alphanumeric characters, stripping multiple whitespaces, stripping punctuation, stripping HTML tags (e.g., ⁇ b>.. . ⁇ /b>), lower casing, stripping numeric (for creating the sparse index), stemming the text (for creating sparse index) and/or other operations.
  • the document dataset builder and indexer 237 may further process the document datasets to build indexes.
  • the indexes may comprise sparse index and dense index.
  • building sparse index may comprise traversing the preprocessed sections and obtain information about the occurrences of each word in each passage/section.
  • building sparse index may comprise collecting token statistics to compute term frequency-inverse document frequency (TFIDF) values for each token in each passage.
  • TFIDF term frequency-inverse document frequency
  • the sparse index may be used to determine the saliency the word/term in a passage. For example, the sparse index may be computed based on a count of occurrences of a term in the portion of the passage and an inverse document frequency (IDF) weight.
  • IDF inverse document frequency
  • the IDF weight may represent how common or rare a word is across the sections in the corpus. For instance, the closer it is to 0, the more common a word is.
  • the metric may be calculated by taking the total number of sections, dividing it by the number of sections that contain a word, and calculating the logarithm. It should be noted that any other suitable methods can be utilized to calculate the IDF.
  • the dense index may be used to retrieve the representation (semantic meaning) of a section or any other data structure.
  • Each section may be indexed and may be retrieved by a dense index.
  • a dense index may correspond to a dense representation of a passage/section such as a tensor of floats computed to capture the semantic meaning of the section. Any suitable metrics may be utilized for determining the semantic meaning of a passage.
  • a dense index may be computed by iterating through the tokens for each section, and if the TFIDF value of the token is below a threshold, then prune the corresponding term, otherwise keep the term.
  • the threshold may be fixed (e.g., 0.15) or dynamic.
  • the threshold may be dynamically determined from the distribution of TFIDF values such as the mean of the TFIDF values throughout the entire document or across one or more sections.
  • the word’s embedding vector may be retrieved and scaled the values by the word’s TFIDF value.
  • the scaled word embedding vectors may then be stacked up to form a matrix (e.g., matrix of size len_pruned_passage x word embedding dimension).
  • the dense index may comprise a set of these matrices.
  • Each section or passage may have a dense index.
  • the document dataset and indexes for one or more uploaded documents may be built by the document dataset builder and indexer 237 and stored in the storage 240.
  • the storage 240 may store various other information or data produced or collected during the information retrieval process.
  • the backend component 230 may process the user input and return the information retrieval result to the frontend component. For example, the backend component may identify relevant texts in one or more sections in the original document containing answer to the question (if the user input query comprises a question) or containing relevant information and pass such information to the frontend, the original documents and the relevant information with visual indicator, as well as texts indicative of answer to the user question, may be rendered at the UI 210 at the front end 220 of the system.
  • the backend component 230 may comprise a retriever-reader paradigm to locate the information in the one or more documents.
  • the backend component 230 may comprise a plurality of components such as a retriever engine 231 capable of ranking the one or more sections of the one or more documents by degree of relevance based on the user input query and the document indexes built as described above.
  • the user input query may be received through one or more input channels or in one or more types (e.g., question, search terms, positive keywords, negative keywords, etc ).
  • the retriever engine 231 may comprise a plurality of models (e g., generative model, pseudo-relevance model, retriever score aggregator, etc.) for computing a plurality of scores using the sparse index of the document dataset and the user input, the dense index of the of the document dataset and the user input, using the positive and negative keywords and the document dataset, respectively, as well as assigning scores indicative of probability of relevance of each section, and/or combining the plurality of scores to compute an “information retrieval score” (e.g., IR_score) for each section.
  • the score may indicate the degree of relevancy.
  • the IR_score may be further aggregated with a score based on an answer identified to question to generate a final relevance score. Details about the information retrieval (IR) pipeline of the retriever engine and the operations and models of the retriever engine are described with respect to FIGs. 3-4.
  • the retriever engine 231 may pass the top ranked sections or most relevant sections (ranked based on the aggregated IR score) to the reader engine 233.
  • the reader engine 233 may further process the top ranked sections to identify the “answer” to the question in each section.
  • the reader engine 233 may comprise a deep neural network trained to extract or retrieve a span/piece of text (e.g., start and end locations of tokens and token offset for the answer, location of an answer in a table, coordinates of the answer on a page) from a section, a passage or a page which contains answer to the question. In some cases, the reader engine 233 may not be trained to answer a given question.
  • the reader engine may be trained to understand/interpret the question and identify a piece of texts in the one or more sections (passed by the retriever engine) that contains the answer to the question.
  • the reader engine 233 may be an extractive-type question answering system that is trained to extract answer from the original document. As described above, the reader engine may output a score based on the answer identified in a section which is further combined with the IR_score associated with the corresponding section to generate a final relevance score. In some cases, the reader engine 233 may also perform continuous learning based on user feedback data.
  • the reader engine may further comprise a generative model such as a generative Large Language Model (LLM) which is trained to generate new text as an answer to a question based on the piece of texts identified from the original document. Details about the reader engine are described later herein.
  • LLM generative Large Language Model
  • the backend component 230 may further comprise a recommender engine 235.
  • the recommender engine 235 may comprise a lightweight model trained to improve the ranking of the sections (based on the information retrieval scores). For instance, the model may be trained to optimize the interpolation of the information retrieval scores generated by the retriever engine.
  • the input features for the recommender model may comprise the plurality of scores from the information retrieval pipeline (e.g., sparse index score, dense index score, generative model score, pseudo relevance score, etc.) and the output may be a combined/improved IR score.
  • the recommender model may be a classifier trained on the clickthrough data of the user indicative whether a given section is relevant.
  • the training datasets may be paired datasets or labeled data generated based on a user clicking on one or more sections indicative of relevancy. This beneficially improves the relevancy score by taking into account the user knowledge or feedback collected during the process.
  • one or more systems or components of the system may be implemented as a containerized application (e g., application container or service containers) as described elsewhere herein.
  • the application container may provide tooling for applications and batch processing, such as web servers with Python or Ruby, JVMs, or even Hadoop or HPC tooling.
  • the frontend of the system may be implemented as a web application using the framework (e.g., Django Python) hosted on an Elastic Cloud Compute (EC2) instance on Amazon Web Services (AWS).
  • the framework e.g., Django Python
  • EC2 Elastic Cloud Compute
  • AWS Amazon Web Services
  • the backend of the system may be implemented as serverless compute service such as hosted on AWS Lambda as a serverless compute service running a web framework for developing RESTful APIs (e.g., FastAPI). This may beneficially allow for a large-scale implementation of the system.
  • the backend system e g., AWS Lambda
  • one or more functions or operations consist with the methods described herein can be provided as software application that can be deployed as a cloud service, such as in a web services model.
  • a cloud-computing resource may be a physical or virtual computing resource (e g., virtual machine).
  • the cloud-computing resource is a storage resource (e.g., Storage Area Network (SAN), Network File System (NFS), or Amazon S3.RTM ), a network resource (e.g., firewall, load-balancer, or proxy server), an internal private resource, an external private resource, a secure public resource, an infrastructure-as-a-service (laaS) resource, a platform-as-a-service (PaaS) resource, or a software-as-a-service (SaaS) resource.
  • a cloud-computing service provided may comprise an laaS, PaaS, or SaaS provided by private or commercial (e.g., public) cloud service providers.
  • the machine learning algorithm may comprise one or more of the following: a support vector machine (SVM), a naive Bayes classification, a linear regression, a quantile regression, a logistic regression, a random forest, a neural network, convolutional neural network (CNN), recurrent neural network (RNN), a gradient-boosted classifier or repressor, or another supervised or unsupervised machine learning algorithm (e.g., generative adversarial network (GAN), Cycle-GAN, etc.).
  • SVM support vector machine
  • CNN convolutional neural network
  • RNN recurrent neural network
  • GAN generative adversarial network
  • Cycle-GAN Cycle-GAN
  • FIG. 3 and FIG. 4 shows examples various components and operations of the information extraction and retrieval system, in accordance with embodiments of the present disclosure.
  • the retriever 310, recommender 330 and reader 320 can be the same as the retriever engine, reader engine and recommender engine as described above.
  • the system may allow users to specify the information of interest/need via one or more input channels or types.
  • the one or more input channels may comprise question 301, search terms 303 and positive and negative keywords 305.
  • a user may choose to provide the query input using one or more of the multiple channels.
  • the “question” channel 301 may permit a user to input a question in natural language (e.g., “what is the governing law?”, “what is the operational risk event predicted for file x being delayed from source system A?”).
  • the question may be processed to determine whether it is a valid question.
  • a valid question may be the one that can be answered by identifying a span/piece of texts in the original document (e.g., “what is the governing law?”).
  • An invalid question may be one that no extractive answer can be identified (e.g., “is the agreement construed under French law?”).
  • the question may be processed regardless the validity. The system may return “no answer” if no extractive answer is identified.
  • the system may comprise a generative model such as a generative LLM to generate answer texts based on the extractive answer. For example, the system may return a generated answer text (e.g., not exactly the same as the original texts) based on the piece of texts provided by the reader engine.
  • the question input 301 may be fed to both the retriever 310 and the reader 320.
  • the question input 301 may be fed to the reader 320 and used by the reader to identify an answer in one or more relevant sections/passages.
  • the question input fed into the retriever 310 may be pre-processed 311 to be tokenized into words and may form part of a unified query.
  • the search term input 303 may be query terms.
  • the one or more search terms may be used to search the relevant information such as based on intent match, phrase match, semantic matching, broad match or others.
  • the system may employ one or more algorithms or rules to interpret the query terms and provide results that match the search intent (e.g., results may not contain the exact term). For instance, latent semantic indexing, co-occurring terms and synonyms or other natural language processing techniques may be utilized to provide the search results.
  • the search terms may be treated regardless the order (e.g., caput mundi and mundi caput will result in the search terms ⁇ caput mundi ⁇ ), and/or may be case insensitive.
  • the search term may be treated as word that is truncated to the root (e.g., “governed”, “governing”, “govern” are identified with “govern”). This beneficially provides a more robust search result by avoiding OCR errors or spelling errors which may occur at the end of a word (e.g., “gorvennnned” is treated as “govern”).
  • the search term may be treated for searching a term has a similar semantic meaning of the search term (e g., search results containing “salary”, “compensation” will be returned for search term of “remuneration”) rather than the literal exact match.
  • the search term 303 and the question 301 may be pre-processed 311 by the retriever to generate a unified query for search.
  • the unified query may be generated based on the question, the search terms or a combination of both depending on the availability. For instance, if the user input query includes only a question, the unified query is constructed based on the question only.
  • the question 301 and the search terms 303 may be tokenized into words and combined into a single, unified query 312 for search.
  • generating the query 312 may comprise de-duplicate terms in the question and the search terms.
  • the unified query 312 may be used to score the sections/passages of the one or more uploaded documents with both the sparse index and the dense index 313.
  • the unified query may be processed for index construction (i.e., building sparse index and dense index for the unified query) for retrieving the query and representation of the query.
  • index construction i.e., building sparse index and dense index for the unified query
  • no index is built for the query.
  • the representation of the query may be compared against the representation of sections of the documents to calculate the similarity score where the section and representation of the sections are retrieved by the sparse index and the dense index respectively.
  • the keywords input 305 may comprise positive keywords, negative keywords or a combination of both.
  • the keywords may be used in a form of distant supervision.
  • the keywords may be used to build “weak” or “soft” labels via aggregation of the labelling functions, indicator functions or signals, such as by majority vote or by a generative model built on top of the labelling functions (e.g., utilizing the labels as emission probability features).
  • the keywords may be used as “weak” or “soft” labels for determining relevancy of query results.
  • the keywords may not be used to construct the unified query.
  • the system may use the presence of a keyword in a section (positive keyword or negative keyword) to determine a relevancy of the section.
  • the section may be identified as a search result based on the question and/or search terms. For example, one or more sections may be identified as relevant based on question and/or search terms, and if a positive keyword is matched with a word in a given section, the section may be labeled as relevant. Similarly, if a negative keyword is matched with a word in a given section, the section may be labeled as irrelevant.
  • the match of the positive keywords or negative keywords may be determined using any suitable algorithms. In some cases, the match may not be exact literal match. For example, the match may be determined based on regex (regular expression) features wherein pattern matching may be utilized to search for particular strings of characters rather than constructing multiple, literal search queries.
  • the keywords may be used to construct indicator functions that may be used to generate training datasets to train a generative model.
  • the keyword may be used as scoring function (e.g., emission probability matrix).
  • the keywords may not be used to form the unified query but a score outputted by the generative model may be aggregated with the sparse index score and dense index score to generate a combined relevancy score for a section.
  • the keywords 305 may be inputted in various formats such as comma-separated lists of words.
  • the system may allow users to specify different spellings of a keyword, conjunctions and disjunctions of words and the like by separating the keywords with or any other suitable symbols, using a DSL (domain specific language) and the like.
  • a user may be required to provide at least one keyword for each type (positive or negative) in order for the keywords to have an effect on the search result. Alternatively, a user may be required to provide a keyword for either type.
  • the retriever 310 may comprise a plurality of models to compute a plurality of scores and aggregate the plurality of scores to output an information retrieval score (IR_score) in a unique process.
  • the retriever pipeline may comprise searches based on the sparse and dense indexes, models trained following a pairwise protocol in a learning-to-rank framework, relevancy predictions by generative models trained using the keywords, and an aggregation of the plurality of scores to return a single information retrieval score.
  • the retriever 310 may perform sparse index scoring based on the sparse index corresponding to the unified query 313 and the sparse index corresponding to the section/passage.
  • the sparse index score 314 may be a lexical match score measuring a lexical match between the query terms and each section terms based on the sparse index.
  • the sparse index score for each section may be computed by finding all matches between the query terms and section terms (after the preprocessing of both the query terms and the section terms) and then weighing each match by how important the matched term is in the corpus of sections. Any suitable lexical match retrievers such as BM25 or other methods such as Boolean Retrieval and statistical language models may be utilized.
  • the retriever may utilize the BM25 as the lexical-match score to determine the similarity (degree of relevancy).
  • the lexical match may advantageously provide improved efficiency. With summation over exact matches, scoring of each query term only goes to documents that contain matching terms. This can be done efficiently using inverted list indexing.
  • the lexical-match score can also be conveniently applied to n-grams.
  • the retriever 310 may also compute a dense index score 315 to measure a similarity between the query word and each section/passage.
  • the dense index score may be a semantic similarity score.
  • the query word may be processed to generate a representation such as a tensor of floats which is computed along with the representations of sections stored in the dense index to compute the similarity.
  • a dense index may be created corresponding to the representation that can be used for lookup of the representation.
  • the system may not generate a dense index for retrieving the query representation.
  • a dense index score may be generated corresponding to an RV score (multivariate generalization of the squared Pearson correlation coefficient) between the query word embedding matrix and word embedding matrix of each passage.
  • RV score multivariate generalization of the squared Pearson correlation coefficient
  • the query matrix (e.g., tensor of floats or dense index) may be constructed similarly to the passage matrix (e.g., dense index for the passage) except that the terms may not be pruned in the unified query.
  • the retriever 310 may also compute a generative model score 317.
  • the generative model score 317 may be computed by an unsupervised probabilistic model which interprets the presence of positive and negative keywords in any given section into a probability that the section is relevant.
  • the generative model score 317 may be the output of a generative model. As described above, the generative model may produce weak labels using the keywords 305.
  • the keywords 319 may be used to construct an indicator function that may return, for each passage, a statement of whether the keyword is found in the passage (a match of the keyword).
  • the indicator function may comprise user-defined rules indicating whether a section is relevant or irrelevant.
  • a user may be permitted to specify different kinds of heuristics or rules to indicate whether a section is likely be relevant or irrelevant.
  • a user may be permitted to specify any other heuristics such as types of data (e.g., date, type of entity, addresses) and the like.
  • the positive/negative keywords received via the user interface may be used by the system to construct the indicator function automatically without requiring a user to define the rules explicitly.
  • the indicator function may return a Relevant label.
  • an Irrelevant label may be returned for the given passage.
  • the retriever may further assign a prior probability of relevance for each passage.
  • the generative model may then be fitted to the entire universe of passages across all uploaded documents.
  • the generative model may be applied to make inference to obtain a posterior probability distribution over the probability of relevance of each passage 317.
  • the output of the generative model may comprise a generative model score 317 in the range [0, 1] assigned for each passage.
  • the retriever 310 may fit and apply a pseudo-relevance model 318 to the sparse index 314, dense index 315 and generative scores 317 to generate a pseudo-relevance score 316.
  • the pseudo-relevance model 316 may be trained using supervised learning and may be a scoring model.
  • the paired datasets or the labels (relevant/not relevant) used to train the model may not be provided by human. Instead, the labels may be automatically generated from ranking based on the other scores as proxies (e.g., BM25 score, RV similarity score, generative model score, etc ).
  • the pseudo relevance score may indicate answer to the question "What caused the other scores to be high (low)?" and may use the patterns (e.g., rankings based on the BM25 score, RV similarity score, generative model score) it learns to assign scores to sections based on what it thinks makes a section relevant.
  • patterns e.g., rankings based on the BM25 score, RV similarity score, generative model score
  • the labels (relevant/not relevant) used to train the model may be automatically generated by: ranking the sections/passages according to the following priority: first by the sparse index score (e g., BM25 score), then by the dense index score (e g., RV similarity score) and finally by the generative model’s predicted probability of relevance (e.g., generative model score).
  • a number (n) of the top ranking passages across all documents e.g., top ranking 10 passage
  • a number (m) of the bottom ranked passages e.g., bottom ranking 100 passages
  • irrelevant pseudo irrelevant
  • the top-n and bottom-m passages as ranked based on the other scores are likely to be approximately linearly separable in TFIDF space and suitable for training a logistic regression model.
  • These passages along with the pseudo labels may form paired datasets (passage, pseudolabel) used for training a logistic regression model.
  • the logistic regression model may be trained using bag of words features (e.g., uni-grams and bi-grams) in the pairwise approach to Leaming-to-Rank.
  • the bag of words (BoW) features may represent the occurrence of words within a document disregarding the order or structure of the words.
  • the PseudoRelevance model once properly trained, may predict, for each passage, the probability of relevance (e.g., pseudo-relevance score 316).
  • the BoW features such as the uni-gram and bigram TFIDF vectors can be good representatives of document content supported by the cluster hypothesis.
  • the plurality of scores comprising the sparse index score 314, dense index score 315, pseudo-relevance score 316 and generative model score 317 may then be combined 349 to produce a single information retrieval score (e.g., IR score).
  • a weighted average may be used to combine the plurality of scores.
  • the information retrieval score can be computed using any suitable formula and/or coefficient to combine the varieties of scores. Alternatively, other methods such as rank-based approaches may be employed to determine the relevancy instead of computing a score.
  • a Relevance Model 331 of the recommender system 330 may be used.
  • the relevance model 331 may be a lightweight model used to optimize the interpolation of the information retrieval scores 333 to improve the relevancy ranking in response to user feedback.
  • the relevance model 331 may be trained to improve the ranking of the sections. For instance, the model may be trained to optimize the interpolation of the information retrieval scores generated by the retriever engine 333. It should be noted that other methods such as ranking-based approach may be used to generate the IR score. For example, IR score may be generated based on the ranks of the other scores such as utilizing Reciprocal-Rank-Fusion.
  • the input features for the relevance model 331 may comprise the scores from the information retrieval pipeline (e.g., sparse index score, dense index score, generative model score, pseudo relevance score).
  • the recommender model may be a classifier trained on the clickthrough data of the user indicative whether a given section is relevant.
  • the training datasets may be paired datasets or labeled data generated based on a user clicking on one or more sections indicative of relevancy.
  • the recommender model may be automatically retrained when positive sections as clicked by a user is greater than a predetermined threshold 348.
  • the retriever may identify a number of most relevant sections (e.g., top n sections ranked by IR_score 335) and documents which are most-likely to contain the relevant information based on the IR score 333.
  • the relevance score of a document may be defined to be equal to the relevance score of its most relevant section. If the user has not specified a Question in the use input, the top n documents and sections may be presented to the user within the GUI in the order of decreasing relevance score. If the user input comprises a Question, the top n relevant sections of the documents may be passed to the reader 320 for further processing.
  • the reader 320 may comprise an extractive question answering model 321.
  • the extractive question answering model 321 may be a deep neural network trained to extract/retrieve a span/piece of text (e g., start and end token offset for highlighting the answer) from a section/passage which contains answer to the question.
  • the extractive question answering model 321 may be trained to generate an answer to a given question by processing the documents.
  • the extractive question answering model 321 may be trained to understand/interpret the question and identify a piece of texts in the one or more sections (passed by the retriever engine) that contains the answer to the question.
  • the extractive question answering model 321 may be an extractive-type question answering system that extract answer from the original document. For example, the extractive question answering model 321 may not return an answer if the question is invalid (e.g., question cannot be answered by identifying spans of texts in the original document).
  • the reader 320 may further comprise a generative Large Language Model (LLM) to generate an answer in natural language.
  • LLM may be running as a service, hosted by the system herein or a third-party system.
  • the service may be hosted by OpenAI, and accessed through the corresponding API.
  • the underlying LLM model may be a GPT-type model, and the API calls may be configured to use the available model (e.g., text-davinci-003 model or gpt-3.5-turbo or any other model that OpenAI or other vendors make available).
  • the LLM service may be built and hosted by the system or platform herein as an integral component.
  • the input to the generative LLM may comprise the piece of texts identified by the retriever engine from the original document and the question.
  • one or more sections or passages identified by the Retriever may be combined with the natural question in a prompt to the generative LLM.
  • the prompt may be specified to instruct the LLM to generate the answer based only on the identified sections or passages and use the same wording as provided in the passages.
  • the prompt may specify in natural language that "Given these passages [insert retrieved passages here], answer the following question [insert question here].
  • the prompt may specify the model’s “confidence” in its predictions to be a higher confidence (e.g., a lower confidence level resulting in more diverse and unexpected text and a higher confidence level resulting in more conservative and predictable text) such as by setting the temperature parameter of the model to 0, in order to get reproducibility of the answer.
  • the generated answer from the LLM may be compared to the identified piece of texts or passages provided in the prompt and a substring from each passage provided as the identified answer based on a fuzzy string matching algorithm.
  • the fuzzy string matching algorithm may identify a match of the generated answer at a certain position in a given passage if the number of characters in agreement between the generated answer and the given passage, when generated answer is placed at that position in the given passage, exceeds a given threshold.
  • the matched texts may be highlighted or displayed with visual indicator so the user can visually verify the supporting texts.
  • the generated answer may be presented to the user directly, with all of the retrieved passages highlighted as 'supporting evidence.
  • an invalid question for an extractive question answering model may still be returned with a generated answer provided by the LLM.
  • question cannot be answered by identifying spans of texts in the original document may still get an answer generated by the LLM based on paragraphs identified by the retriever engine. Displaying the retrieved paragraphs and/or visual indicator of the supporting evidence may beneficially allow a user to visualize and verify the authenticity or authority of the generated answer.
  • the extractive question answering model 321 may perform continuous learning based on user feedback data.
  • the output 323 of the extractive question answering model 321 may comprise an answer (e g., comprising start and end token offset for the spans of texts) and a score (e.g., QA_score) indicative of likelihood of an answer.
  • the extractive question answering model 321 may be a transformer model for processing the question input, document dataset and identifying the answer to the question.
  • the transformer model may a Bidirectional and Auto-Regressive Transformer (BART) which is a Transformer that combines the Bidirectional Encoder (i.e. BERT like) with an autoregressive decoder (i.e.
  • the Bidirectional Encoder Representations from Transformers is a transformer language model with a variable number of encoder layers and self-attention heads.
  • the BERT may be pre-trained on two tasks: language modelling (e g., tokens may be masked and BERT may be trained to predict them from context) and next sentence prediction (BERT may be trained to predict if a chosen next sentence was probable or not given the first sentence).
  • language modelling e g., tokens may be masked and BERT may be trained to predict them from context
  • next sentence prediction e e e e masked and BERT may be trained to predict them from context
  • next sentence prediction e.g., BERT learns contextual embeddings for words.
  • BERT can be finetuned with less resources on smaller datasets such as custom datasets to optimize its performance on specific tasks such as generating opener sentences.
  • the extractive question answering model 321 may be built on top of a BERT large whole word masking model with a question answering head on top (e.g., trainable start vector of answer span and end vector of answer span).
  • the network architecture may comprise a transformer model with a Multi-Head Attention mechanism to compute representations of its input and output without using sequence aligned RNNs or convolution.
  • the model may use stacked self-attention and point-wise, fully connected layers for both the encoder and decoder.
  • the encoder may comprise a stack of N identical layers each with two sub-layers. The first sublayer is a multi-head self-attention mechanism, and the second sub-layer is a simple, position-wise fully connected feed-forward network.
  • a residual connection may be employed around each of the two sub-layers, followed by layer normalization.
  • the decoder may comprise a stack of N identical layers.
  • the decoder may insert a third sub-layer, which performs multihead attention over the output of the encoder stack. Similar to the encoder, residual connections are employed around each of the sub-layers, followed by layer normalization.
  • the self-attention sub-layer in the decoder stack may be modified to prevent positions from attending to subsequent positions. The multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions.
  • An attention function may map a query and a set of key -value pairs to an output, where the query, keys, values, and output are all vectors.
  • the output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key.
  • the input consists of queries and keys of dimension dk, and values of dimension dv.
  • the multi -head attention may linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and d v dimensions, respectively. On each of these projected versions of queries, keys and values the attention function is performed in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values.
  • the input tokens/output tokens may be converted to embeddings such as vectors of dimension dmodei.
  • a learned linear transformation and softmax function may be used to convert the decoder output to predicted next-token probabilities.
  • the model for extracting unsupervised features may be refined or tuned as new data is collected from the system.
  • the generative LLM may be running as a service, hosted by the system herein or a third-party system.
  • the service may be hosted by OpenAI, and accessed through the corresponding API.
  • the underlying LLM model may be a GPT-type model, and the API calls may be configured to use the available model (e.g., text-davinci-003 model or gpt-3.5-turbo or any other model that OpenAI or other vendors make available).
  • the LLM service may be built and hosted by the system or platform herein as an integral component.
  • the model may comprise a sparse autoencoder with an RNN (recurrent neural network) architecture, such as LSTM (long-short-term memory) network, may be trained to regenerate the question input and document data.
  • RNN recurrent neural network
  • LSTM long-short-term memory
  • an encoder-decoder LSTM model with encoder and decoder layers may be used to recreate a low-dimensional representation of the input data (i.e., low-dimensional features) to the following model training despite a latent/hidden layer.
  • the unsupervised features e.g., low-dimensional representation
  • the extracted features may be used to train two or more classifiers, which accept the extracted features as inputs and map them to the appropriate class label.
  • the language model may be pre-trained on a large dataset.
  • the complete reader model may be fine-tuned on the dataset comprising question-passage pairs (e.g., ⁇ 100k question-passage pairs) for the question answering head.
  • the model’s pre-trained checkpoint may be loaded to the system then the model is further trained on the system’s Treasure Chest data (e.g., ⁇ 25k question-passage pairs).
  • the treasure chest data may be data which have not been used to train the model.
  • the weighted Fl score on the Treasure Chest test set is about 86% over 46 question “types” (e.g., governing law, agreement date, parties, etc.).
  • the system herein may comprise sockets to integrate the variety of machine learning trained models and/or functional components.
  • a socket to a GPU API may be provided for time-consuming or computational expensive models.
  • Other less-computationally expensive processes may be implemented as parallel CPU processing steps.
  • a socket may provide a programming construct, an instance, or instantiate, that can make use of suitable protocol to send and receive data.
  • a socket may be Web Socket APIs allow bi-directional, full-duplex communication between clients and servers. The API may not require a new connection to be set up for each message to be sent between clients and servers. Once the connection is set up the data can be sent and received continuously without any interruption.
  • Sockets such as WebSocket APIs are suitable for Applications with low latency or high throughput requirements.
  • various functional components of the system herein may be seamlessly integrated into a third-party platform or on-premises environment via customized Software Development Kit (SDK).
  • SDK Software Development Kit
  • the QA_score, answers identified by the reader 320 and the IR_score 333 generated by the recommender 330 or the retriever 310 may be processed by a relevance score calculator 340 to generate a final relevance score 341.
  • the relevance score calculator 340 may utilize any suitable equation or algorithm to calculate the final relevance score. For instance, the final relevance score associated with a section may be calculated based on both the corresponding IR score, and the score (e.g., QA score) indicative of the presence of an answer identified in the section.
  • An example of the method for calculating the final relevance score is the following: IR_score + (-1) A has_answer * QA_score.
  • the user may provide feedback 344 which may be used to refine, and retrain the models.
  • the models may continuously learn and improve based on the user feedback data.
  • the user feedback 345 may indicate approval or rejection of relevant information, relevant passage and/or answer identified by the system. For example, a user may be presented a number of top ranked sections within the GUI.
  • the GUI may provide a view of the document in a predefined format (e.g., PDF view), on a first panel allowing the user to navigate the properly rendered document and to annotate (e g., via highlighting, underline, or using any visual indicator) any span of texts in the document.
  • a number of the top ranked passages may be shown.
  • the number of identified relevant passages to be shown may be configurable. The user may inspect the suggested passages, and a click on any of these passages may redirect the PDF view to the passage in the document.
  • the GUI may display both the relevant information and the answer using different visual indicators such that such information is easily identifiable by a user.
  • the relevant information in each of these passages may be rendered in a first color (e g., yellow) on the PDF view.
  • a Question is specified in the user input
  • one or more of the passages may further contain an answer to the question annotated/highlighted in a second color (e.g., blue).
  • any suitable visual indicator e.g., any color for highlighting, underline, markup, etc.
  • the GUI may provide features for collecting user feedback. For example, graphical elements adjacent to each of the suggested answers (e.g., Confirm/Cancel button) may allow the user to confirm or cancel the proposed annotation. In some cases, the user may also be permitted to annotate the relevant span of text directly if an answer has not been found.
  • the relevance feedback may be stored and updated in the relevance feedback store 346.
  • the user feedback may also include information based on a user action or user interaction with the identified sections. For example, any passage or answer that has been proposed as relevant but the user has not explicitly confirmed may be treated by the system as not relevant to the user's information needs. Any passage which has not been presented to the user (e.g., not contained within the top 5 ranked passages) may be treated by the system as un-annotated (i.e., neither relevant nor irrelevant). In some cases, user actions such as how quickly or the time takes for a user clicks on the suggested information/answer may also be part of the user feedback indicating the relevancy of the retrieved information.
  • the user feedback collected by the system herein may indicate relevancy of a passage or answer retrieved by the system herein.
  • the relevancy may be binary (e.g., relevant or not relevant) or multi-level (e g., multiclass, multi-label, or grade indicating relevancy level).
  • Such user feedback may be utilized to further tune/train the models for information extraction and models in the retriever-reader pipelines (e.g., generative model, pseudo-relevance model, relevance model, extractive question answering model, etc.) and/or validate the performance of the aforementioned models (e.g., used for model validation).
  • the user generated relevance feedback may be used to improve the ranking of passages and the relevance of the proposed sections.
  • the model e.g., relevance model 331
  • Positive passages may refer to passages identified by the system as relevant to the user’s query input. For example, at least 15 positive passages may be collected and used to train a model to identify sections of relevance.
  • the model may be retrained when at least 5 additional positive sections (e.g., equal to or greater than a retrain threshold of 5 positive passages) have been accumulated in addition to the initial 15 passages.
  • the retraining may be triggered and executed automatically in the backend component without user intervention.
  • the retrained model may be deployed and may make a prediction for each section to update the relevance of that passage to the user’s interest.
  • the prediction may be a probability that the given passage is of interest to the user and the ranking is based on that probability of relevance.
  • FIGs. 5-10 show examples of user interfaces provided by a frontend component of the system, in accordance with some embodiments of the present disclosure.
  • FIG. 5 shows an example of a GUI for a user to input query for searching information/insight in one or more uploaded documents.
  • a user may upload one or more documents to be analyzed by the system for information retrieval.
  • the UI may provide a landing page where the user can upload one or more documents into the platform to start a session for information retrieval.
  • a user may be permitted to upload a new document, remove a document and the like via the document upload UI.
  • a user may provide the document by drag/drop, providing link to the document via the UI and may click a graphical element (e.g., graphical button “Upload Documents”) displayed within the UI to upload the document.
  • the system may automatically process the document to build document dataset and indexes as described above.
  • the user may input query for searching information/insight in the one or more uploaded documents via the GUI comprising multiple input channels as illustrated in FIG. 5.
  • the multiple input channels may allow the user to specify the interested/needed information in multiple forms. For example, a user may be allowed to provide an intelligent question 501, search terms 503, positive keyword 505, negative keywords 507 or any combination of the above to specify the desired information.
  • the input channels may permit a user to provide any other types of inputs (e.g., data types such as dates, addresses, etc.) related to a query as described elsewhere herein.
  • the system may determine the type of question and select or adapt the backend models according to the type of question accordingly. For instance, the system may utilize more regexes (regular expressions) if the user inputs indicate the data types are dates.
  • the GUI may allow a user to provide inputs to the multiple input channels in any suitable forms.
  • the input may be provided via a text field each corresponding to an input channel for direct text inputs as illustrated in FIG. 5.
  • the GUI may or may not display system suggested search terms, questions, or the keywords.
  • the system may display suggested question or complete a question as the user typing the question.
  • the user input may be provided for at least one of the input channels and may be processed by the system as described elsewhere herein.
  • the “question” field 501 may receive a question in natural language (e.g., “what is the governing law?”, “what is the operational risk event predicted for file x being delayed from source system A?”).
  • the question may be received by the backend component and processed to determine whether it is a valid question.
  • a valid question may be the one that can be answered by identifying a span/piece of texts in the original document (e.g., “what is the governing law?”).
  • An invalid question may be one that no extractive answer can be identified (e.g., “Is the agreement construed under French law?”) when the reader engine comprises only the extractive model not the LLM.
  • the GUI may prompt the user to enter a new question when an invalid question is determined or display “no answer”.
  • the question may be received and processed by the backend component and used to identify an answer regardless the validity.
  • An invalid question may result in “no answer” displayed to the user within the GUI or only the relevant information being highlighted in the original document.
  • an invalid question for an extractive question answering model may still be returned with a generated answer provided by the LLM. For example, question cannot be answered by identifying spans of texts in the original document may still get an answer generated by the LLM based on paragraphs identified by the retriever engine.
  • the search terms input field 503 may receive query terms inputted by a user.
  • the search terms may be inputted in the text field 503 in various formats such as comma-separated lists of words.
  • the positive keyword input field 505 and negative keyword input field 507 may receive the keywords inputted in various formats such as comma-separated lists of words.
  • the system may allow users to specify different spellings of a keyword such as by separating the keywords with
  • the user may provide input for at least one of the multiple channels.
  • the user may start a search session by clicking on the “Rank Sections” 601 button.
  • the GUI may show the results to the user.
  • the GUI may provide a first panel 610 showing the top n ranked passages (e.g., passages 605, 607) as most relevant to the user input query. For example, up to five passages from the uploaded document may be displayed in the first panel 610 as the search result and ranked based on the relevancy.
  • Each passage may contain relevant information identified by the system. Displaying multiple relevant passages in the first panel may allow the user to conveniently inspect and compare multiple passages. A user may be permitted to select any of the passages such as by clicking on a passage 607.
  • a view of the document (e.g., PDF view) with the user selected passage may be displayed in a second panel 620.
  • the second panel 620 may render the document in a pre-defined format (e.g., PDF view) and may update a rendering of a portion of the document upon a user selecting a different passage in the first panel 610.
  • the second panel 620 may allow a user to navigate the rendered document to annotate relevant information (determined by the user) such as by highlighting, any span of texts in the document.
  • relevant information determined by the user
  • any suitable features can be employed for a user to provide or visualize the relevancy information. For example, a heatmap showing degree of presumed relevance may be displayed within the GUI.
  • the user may navigate within a document (e.g., moving pages) and/or navigate across different documents (e.g., clicking on “next document” 627).
  • any annotation generated by the system and/or provided by the user may be stored before the next document is rendered.
  • the user may annotate any texts in the rendered document indicating relevant or not relevant and such input may be utilized as user feedback to further improve the models in the backend component as described above.
  • the passage identified as relevant 621 may be highlighted in a first color (e g., yellow). If the user input includes a question, the highlighted one or more passages may further contain an answer to the question annotated/highlighted in a second color (e.g., blue) 623.
  • the second panel may provide graphical elements adjacent to each of the suggested answers (e.g., buttons of Confirm/Cancel 625) allowing the user to confirm or cancel the proposed answer annotation.
  • FIG. 7 shows another example of the GUI for a user to provide feedback by confirming or rejecting a system identified answer.
  • a passage containing relevant information may be displayed to the user.
  • the relevant information may be highlighted in a first color (e.g., yellow).
  • the answer may be hi hlighted/annotated in a second color 703, such that the answer is easily identifiable by the user.
  • the highlighted relevant information portion 701 may be longer, for example, a clause of a legal document, referred to a ‘chunks’ of text.
  • the relevant information portion may contain the answer portion 703, for example, a word or phrase, that directly answers the question.
  • a word or phrase when a word or phrase is an answer to a question, it may also form part of a chunk of text which is identified as relevant information based on the search terms or a combination of the search terms and the question.
  • the GUI may permit a user to provide a feedback indicating whether the extracted answer 703 is relevant or not by clicking on the button 705 to confirm or cancel the extracted answer.
  • the GUI only illustrates the graphical elements for providing feedback regarding the answer, a user may also be allowed to provide feedback regarding the relevancy of the relevant information 701 in a similar manner.
  • the GUI may allow a user to change or update a search at any point during the process.
  • a search may be updated by modifying a query input (e.g., modifying search terms, questions, keywords, adding/removing an input to an input channel) and/or by uploading/removing a document to be analyzed.
  • the user may click on the “clear session” button 801 to search for new information across the same one or more documents (e.g., entering a new question) or uploading one or more new documents.
  • the “clear session” may trigger a drop-down menu allowing the user to select to modify a query or modify the underlying documents.
  • the learned information for the session may be cleared.
  • the learned information may be reserved when a user intends to search across multiple sessions.
  • the GUI may allow a user to refine a search at any point during the process. Refining a search may not start a new search session. For instance, instead of clicking on the “clear session” button, the user may modify the search terms, questions, keywords, adding/removing an input to a selected input channel to refine the input query. The refined input query may be applied to the uploaded documents. All the previous annotations generated by the system or provided by the user may be persisted (e.g., stored in a memory/storage, or database). In some cases, the user may not be shown previously identified passages.
  • the GUI may allow users to export the annotated information. For example, once a user is satisfied with having annotated a sufficient number of documents/passage, the annotations (including the relevant information and/or answer, or top N ranked relevant sections) may be exported to a file in a selected format (e.g., .json file) by clicking on the “export data” button 803.
  • a user may export and save the extracted information such as by specifying the exported document type, the name of the insight, the question number, the type of extraction and the like.
  • the user may also be permitted to share and publish the insight via the platform such that the insights may be searchable by the user and/or other users.
  • the created insight may be added to a knowledge database that may not be shared with other users.
  • the knowledge base may be shared with other users with access permission.
  • the user may provide inputs in one or more of the multiple channels.
  • the user may be required to provide an input in at least one of the input channels.
  • the input query may comprise any combination of the multiple channels.
  • FIG. 8 shows an example when the input query contains only search terms. In this case, the extracted relevant information may not comprise an answer whereas the extracted relevant information may be based on the search terms only.
  • FIG. 9 shows an example where the input query contains only a question. In the illustrated example, the extracted information comprises a word or phrase as an answer to a question, which also forms part of a chunk of text which is identified as relevant information based on a unified query generated according to the question.
  • FIG. 10 shows an example where the input query comprises both a question and search terms.
  • aspects of the systems and methods provided herein can be embodied in programming.
  • Various aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of machine (or processor) executable code and/or associated data that is carried on or embodied in a type of machine readable medium.
  • Machine-executable code can be stored on an electronic storage unit, such as memory (e.g., read-only memory, random-access memory, flash memory) or a hard disk.
  • “Storage” type media can include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming.
  • All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer into the computer platform of an application server.
  • another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links.
  • the physical elements that carry such waves, such as wired or wireless links, optical links or the like, also may be considered as media bearing the software.
  • terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.
  • a machine readable medium such as computer-executable code
  • a tangible storage medium such as computer-executable code
  • Non-volatile storage media include, for example, optical or magnetic disks, such as any of the storage devices in any computers) or the like, such as may be used to implement the databases, etc. shown in the drawings.
  • Volatile storage media include dynamic memory, such as main memory of such a computer platform
  • Tangible transmission media include coaxial cables; copper wire and fiber optics, including the wires that comprise a bus within a computer system.
  • Carrier-wave transmission media may take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) and infrared (IR) data communications.
  • RF radio frequency
  • IR infrared
  • Common forms of computer-readable media therefore include for example: a floppy disk, a flexible disk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any other optical medium, punch cards paper tape, any other physical storage medium with patterns of holes, a RAM, a ROM, a PROM and EPROM, a FLASH-EPROM, any other memory chip or cartridge, a carrier wave transporting data or instructions, cables or links transporting such a carrier wave, or any other medium from which a computer may read programming code and/or data.
  • Many of these forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a processor for execution.
  • the present disclosure provides methods and systems for querying insight information.
  • the method comprises: providing different input fields on a UI for receiving a query input for searching insight information in a document, the input fields comprise a first input field for receiving one or more questions in natural language and a second input field for receiving one or more search terms; receiving the query comprising a question or a search term; identifying, using one or more machine learning algorithm trained models, one or more sections of the document each comprising a chunk of texts relevant to the query input, and rendering the one or more sections in the UI with the chunk of texts annotated with a visual indicator; receiving a user feedback data indicating an acceptance or rejection of the chunk of texts; and improving at least one of the trained models using the user feedback data.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Databases & Information Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Artificial Intelligence (AREA)
  • Human Computer Interaction (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

La présente divulgation concerne des procédés et des systèmes d'interrogation d'informations d'aperçu. Le procédé consiste à : fournir différents champs d'entrée sur une UI pour recevoir une entrée d'interrogation pour rechercher des informations d'aperçu dans un document, les champs d'entrée comprenant un premier champ d'entrée pour recevoir une ou plusieurs questions dans un langage naturel et un second champ d'entrée pour recevoir un ou plusieurs termes de recherche ; recevoir l'interrogation comprenant une question ou un terme de recherche ; identifier, à l'aide d'un ou de plusieurs modèles entraînés d'algorithme d'apprentissage automatique, une ou plusieurs sections du document comprenant chacune un bloc de textes pertinents pour l'entrée d'interrogation, et restituer la ou les sections dans l'UI avec le bloc de textes annoté avec un indicateur visuel ; recevoir des données de rétroaction d'utilisateur indiquant une acceptation ou un rejet du bloc de textes ; et améliorer au moins l'un des modèles entraînés à l'aide des données de rétroaction d'utilisateur.
PCT/GB2023/051519 2022-06-13 2023-06-12 Procédés et systèmes de transformation et de récupération d'informations à partir de données de document à l'aide d'un apprentissage automatique Ceased WO2023242540A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202263351534P 2022-06-13 2022-06-13
US63/351,534 2022-06-13

Publications (1)

Publication Number Publication Date
WO2023242540A1 true WO2023242540A1 (fr) 2023-12-21

Family

ID=86904241

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2023/051519 Ceased WO2023242540A1 (fr) 2022-06-13 2023-06-12 Procédés et systèmes de transformation et de récupération d'informations à partir de données de document à l'aide d'un apprentissage automatique

Country Status (1)

Country Link
WO (1) WO2023242540A1 (fr)

Cited By (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117893341A (zh) * 2024-01-12 2024-04-16 上海薪湾科技有限公司 全球薪酬管理方法、系统、电子设备及介质
CN117951272A (zh) * 2024-01-19 2024-04-30 巨翊科技（上海）有限公司 一种基于大语言模型的文档生成方法、系统和介质
CN118069852A (zh) * 2024-04-22 2024-05-24 数据空间研究院 一种多模型融合的数据分类预测方法与系统
CN118135592A (zh) * 2024-05-09 2024-06-04 支付宝(杭州)信息技术有限公司 一种基于医疗llm模型的用户服务方法、装置
CN118277521A (zh) * 2024-02-27 2024-07-02 福建亿榕信息技术有限公司 一种基于llm的电力领域智能问答方法、系统、设备和介质
CN118332072A (zh) * 2024-06-13 2024-07-12 山东浪潮科学研究院有限公司 一种基于rag技术的智能文档检索生成方法和系统
CN118410522A (zh) * 2024-07-01 2024-07-30 蚂蚁科技集团股份有限公司 用于大语言模型的隐私数据保护方法、装置、介质及设备
CN118446338A (zh) * 2024-07-04 2024-08-06 阿里巴巴（中国）有限公司 模型训练方法、数据处理方法、电子设备及存储介质
CN118445379A (zh) * 2024-06-27 2024-08-06 蚂蚁科技集团股份有限公司 大语言模型的训练方法及装置
CN118643144A (zh) * 2024-08-19 2024-09-13 山东浪潮科学研究院有限公司 一种通用自适应问答方法、系统、存储介质及电子设备
CN118760760A (zh) * 2024-09-06 2024-10-11 阿里云飞天(杭州)云计算技术有限公司 面向知识库的冲突内容检测方法、装置、电子设备及介质
CN118939782A (zh) * 2024-10-14 2024-11-12 浙江工商大学 一种基于图数据库的rag存储和检索方法及装置
CN118964541A (zh) * 2024-07-02 2024-11-15 铯镨科技有限公司 建筑知识问答系统和建筑知识答复信息生成方法
CN119106686A (zh) * 2024-11-08 2024-12-10 江苏省质量和标准化研究院 一种基于深度学习的术语处理方法
CN119311880A (zh) * 2024-12-09 2025-01-14 之江实验室 一种多模态文献数据提取方法、装置及介质
CN119377342A (zh) * 2024-12-27 2025-01-28 北京师范大学珠海校区 教学互动内容生成方法、装置、设备及计算机程序产品
CN119599015A (zh) * 2024-11-18 2025-03-11 东南大学 基于大小模型交互的无监督开放关系抽取文本处理方法
US20250103826A1 (en) * 2023-09-21 2025-03-27 Google Llc Processing documents in cloud storage using query embeddings
CN119739714A (zh) * 2024-11-29 2025-04-01 国网福建省电力有限公司电力科学研究院 基于llm动态文本划分的rag知识库构建方法
CN119848223A (zh) * 2025-03-21 2025-04-18 中国人民解放军63921部队 一种针对结构化长文档的问答方法及系统
CN119884309A (zh) * 2024-12-24 2025-04-25 深圳市现代农业装备研究院 基于大语言模型的知识增强问答方法及电子设备
CN120011524A (zh) * 2025-04-18 2025-05-16 深圳唯爱智云科技有限公司 基于大语言模型的正则表达式生成方法及装置
US20250200399A1 (en) * 2023-08-07 2025-06-19 Trunk Tools, Inc. Methods and systems for generative question answering for construction project data
EP4579481A1 (fr) * 2023-12-28 2025-07-02 Atlassian Pty Ltd Interface générative pour contenu multiplateforme
WO2025144846A1 (fr) * 2023-12-26 2025-07-03 Equifax, Inc. Modèle d'intelligence artificielle pour commander un conflit d'interaction
US12361204B1 (en) * 2024-11-06 2025-07-15 Tiny Fish Inc. Extracting relevant information from a document
US20250238604A1 (en) * 2024-01-23 2025-07-24 Docusign, Inc. Document structure extraction
JP2025117087A (ja) * 2024-01-30 2025-08-12 株式会社三菱総合研究所 情報処理装置及び情報処理方法
EP4610908A1 (fr) * 2024-03-01 2025-09-03 Hitachi, Ltd. Système de circulation de connaissances de défauts
WO2025181561A1 (fr) * 2024-02-29 2025-09-04 云智能资产控股（新加坡）私人股份有限公司 Procédé de traitement de réponse à une question, dispositif, produit et support de stockage
CN120596647A (zh) * 2025-08-11 2025-09-05 浪潮通用软件有限公司 一种合同检索增强优化方法、设备及介质
WO2025207839A1 (fr) * 2024-03-25 2025-10-02 Forgen Ai, Llc Génération de message par un grand modèle de langage (llm)
WO2025236899A1 (fr) * 2024-05-15 2025-11-20 阿里巴巴（中国）有限公司 Procédé et appareil de traitement de données, et procédé et appareil de traitement de données de document
WO2025255733A1 (fr) * 2024-06-12 2025-12-18 Microsoft Technology Licensing, Llc Marquage de document externe et augmentation de requête pour génération de modèle de langage
US12566881B2 (en) 2023-04-24 2026-03-03 Palantir Technologies Inc. Data permissioned language model document search
US12602698B1 (en) * 2025-04-29 2026-04-14 Isidore S. Sobkowski Transformation of non-compliant stimulus/response system to operational governance compliant ethical artificial intelligence through the use of dynamic governance guiderails directives and executor
US12608562B2 (en) 2023-09-21 2026-04-21 Google Llc Providing personalized prompts to users based on documents in cloud storage

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180365324A1 (en) * 2017-04-24 2018-12-20 Considdr Inc. Method of data organization and data searching for use in constructing evidence-based beliefs

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180365324A1 (en) * 2017-04-24 2018-12-20 Considdr Inc. Method of data organization and data searching for use in constructing evidence-based beliefs

Cited By (38)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12566881B2 (en) 2023-04-24 2026-03-03 Palantir Technologies Inc. Data permissioned language model document search
US20250200399A1 (en) * 2023-08-07 2025-06-19 Trunk Tools, Inc. Methods and systems for generative question answering for construction project data
US12608562B2 (en) 2023-09-21 2026-04-21 Google Llc Providing personalized prompts to users based on documents in cloud storage
US20250103826A1 (en) * 2023-09-21 2025-03-27 Google Llc Processing documents in cloud storage using query embeddings
WO2025144846A1 (fr) * 2023-12-26 2025-07-03 Equifax, Inc. Modèle d'intelligence artificielle pour commander un conflit d'interaction
EP4579481A1 (fr) * 2023-12-28 2025-07-02 Atlassian Pty Ltd Interface générative pour contenu multiplateforme
CN117893341A (zh) * 2024-01-12 2024-04-16 上海薪湾科技有限公司 全球薪酬管理方法、系统、电子设备及介质
CN117951272A (zh) * 2024-01-19 2024-04-30 巨翊科技（上海）有限公司 一种基于大语言模型的文档生成方法、系统和介质
US20250238604A1 (en) * 2024-01-23 2025-07-24 Docusign, Inc. Document structure extraction
JP7726465B2 (ja) 2024-01-30 2025-08-20 株式会社三菱総合研究所 情報処理装置及び情報処理方法
JP2025117087A (ja) * 2024-01-30 2025-08-12 株式会社三菱総合研究所 情報処理装置及び情報処理方法
CN118277521A (zh) * 2024-02-27 2024-07-02 福建亿榕信息技术有限公司 一种基于llm的电力领域智能问答方法、系统、设备和介质
WO2025181561A1 (fr) * 2024-02-29 2025-09-04 云智能资产控股（新加坡）私人股份有限公司 Procédé de traitement de réponse à une question, dispositif, produit et support de stockage
EP4610908A1 (fr) * 2024-03-01 2025-09-03 Hitachi, Ltd. Système de circulation de connaissances de défauts
WO2025207839A1 (fr) * 2024-03-25 2025-10-02 Forgen Ai, Llc Génération de message par un grand modèle de langage (llm)
CN118069852A (zh) * 2024-04-22 2024-05-24 数据空间研究院 一种多模型融合的数据分类预测方法与系统
CN118135592A (zh) * 2024-05-09 2024-06-04 支付宝(杭州)信息技术有限公司 一种基于医疗llm模型的用户服务方法、装置
WO2025236899A1 (fr) * 2024-05-15 2025-11-20 阿里巴巴（中国）有限公司 Procédé et appareil de traitement de données, et procédé et appareil de traitement de données de document
WO2025255733A1 (fr) * 2024-06-12 2025-12-18 Microsoft Technology Licensing, Llc Marquage de document externe et augmentation de requête pour génération de modèle de langage
CN118332072A (zh) * 2024-06-13 2024-07-12 山东浪潮科学研究院有限公司 一种基于rag技术的智能文档检索生成方法和系统
CN118445379A (zh) * 2024-06-27 2024-08-06 蚂蚁科技集团股份有限公司 大语言模型的训练方法及装置
CN118410522A (zh) * 2024-07-01 2024-07-30 蚂蚁科技集团股份有限公司 用于大语言模型的隐私数据保护方法、装置、介质及设备
CN118964541A (zh) * 2024-07-02 2024-11-15 铯镨科技有限公司 建筑知识问答系统和建筑知识答复信息生成方法
CN118446338A (zh) * 2024-07-04 2024-08-06 阿里巴巴（中国）有限公司 模型训练方法、数据处理方法、电子设备及存储介质
CN118643144A (zh) * 2024-08-19 2024-09-13 山东浪潮科学研究院有限公司 一种通用自适应问答方法、系统、存储介质及电子设备
CN118760760A (zh) * 2024-09-06 2024-10-11 阿里云飞天(杭州)云计算技术有限公司 面向知识库的冲突内容检测方法、装置、电子设备及介质
CN118939782A (zh) * 2024-10-14 2024-11-12 浙江工商大学 一种基于图数据库的rag存储和检索方法及装置
US12361204B1 (en) * 2024-11-06 2025-07-15 Tiny Fish Inc. Extracting relevant information from a document
CN119106686A (zh) * 2024-11-08 2024-12-10 江苏省质量和标准化研究院 一种基于深度学习的术语处理方法
CN119599015A (zh) * 2024-11-18 2025-03-11 东南大学 基于大小模型交互的无监督开放关系抽取文本处理方法
CN119739714A (zh) * 2024-11-29 2025-04-01 国网福建省电力有限公司电力科学研究院 基于llm动态文本划分的rag知识库构建方法
CN119311880A (zh) * 2024-12-09 2025-01-14 之江实验室 一种多模态文献数据提取方法、装置及介质
CN119884309A (zh) * 2024-12-24 2025-04-25 深圳市现代农业装备研究院 基于大语言模型的知识增强问答方法及电子设备
CN119377342A (zh) * 2024-12-27 2025-01-28 北京师范大学珠海校区 教学互动内容生成方法、装置、设备及计算机程序产品
CN119848223A (zh) * 2025-03-21 2025-04-18 中国人民解放军63921部队 一种针对结构化长文档的问答方法及系统
CN120011524A (zh) * 2025-04-18 2025-05-16 深圳唯爱智云科技有限公司 基于大语言模型的正则表达式生成方法及装置
US12602698B1 (en) * 2025-04-29 2026-04-14 Isidore S. Sobkowski Transformation of non-compliant stimulus/response system to operational governance compliant ethical artificial intelligence through the use of dynamic governance guiderails directives and executor
CN120596647A (zh) * 2025-08-11 2025-09-05 浪潮通用软件有限公司 一种合同检索增强优化方法、设备及介质

Similar Documents

Publication Publication Date Title
WO2023242540A1 (fr) Procédés et systèmes de transformation et de récupération d'informations à partir de données de document à l'aide d'un apprentissage automatique
US12189691B2 (en) Natural language question answering systems
US12259879B2 (en) Mapping natural language to queries using a query grammar
US11900064B2 (en) Neural network-based semantic information retrieval
US20240303496A1 (en) Exploiting domain-specific language characteristics for language model pretraining
US11080340B2 (en) Systems and methods for classifying electronic information using advanced active learning techniques
US12038935B2 (en) Systems and methods for mapping a term to a vector representation in a semantic space
WO2020224097A1 (fr) Procédé et dispositif de recommandation de documents sémantiques intelligents, et support de stockage lisible par ordinateur
US20150039536A1 (en) Clarification of Submitted Questions in a Question and Answer System
US20210034621A1 (en) System and method for creating database query from user search query
CN113609847B (zh) 信息抽取方法、装置、电子设备及存储介质
WO2021237082A1 (fr) Récupération d'informations sémantiques fondée sur un réseau neuronal
WO2024173577A1 (fr) Récupération d'informations de corpus liés aux brevets avec des grands modèles de langage sémantique
EP4702467A1 (fr) Recherche sémantiques et résumage pour documents électroniques
WO2023242543A1 (fr) Procédés et systèmes de détermination de l'exactitude d'une sortie d'un modèle d'apprentissage automatique
US20260099533A1 (en) Raw Content Storage and Analysis Using Topic Maps
US12602415B1 (en) Inter-document search using knowledge-enriched vectors
US20260064698A1 (en) Multimodal Data Ingestion And Retrieval For Agent Systems
Ghali Leveraging Generative AI and in Context Learning to Reshape Human-Text Interaction: A Novel Paradigm for Information Retrieval, Named Entities Extraction, and Database Querying
Khlifi et al. Prompt-Driven Knowledge Retrieval in Arabic Medical Agents via Graph-RAG and LLM
Kumar et al. Optimal Query Expansion Based on Hybrid Group Mean Enhanced Chimp Optimization Using Iterative Deep Learning. Electronics 2022, 11, 1556
Maťašová et al. Implementing AI techniques for efficient searching in document databases
Kumar Contextualization of Web contents through semantic enrichment from linked open data

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23733416

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205 DATED 26/03/2025)

122 Ep: pct application non-entry in european phase

Ref document number: 23733416

Country of ref document: EP

Kind code of ref document: A1