Problem Statement:

Finding relevant information efficiently within large document repositories can be a daunting task for businesses, especially when dealing with diverse formats and massive volumes of data. Traditional keyword-based search engines often struggle to deliver precise results, leading to frustration and inefficiency for users.

Input:

The input to our semantic search engine consists of user queries entered into the search interface, along with a vast corpus of documents stored in various formats such as PDFs, Word, and Excel files. These documents, spanning several terabytes, contain valuable information that users seek to access and extract relevant sections from.

Output:

The output of our solution is a set of near-matching documents retrieved from the corpus, along with relevant sections extracted based on the user’s query. By leveraging advanced NLP techniques, our search engine enhances the user query experience by providing real-time, low-latency results that closely match the users’ information needs.

Challenges Faced:

One of the main challenges in this project was efficiently searching through a large corpus of documents while maintaining real-time performance and low latency. Additionally, extracting relevant sections from documents in various formats posed a significant challenge, requiring robust NLP techniques to accurately identify and extract the desired content.

Proposed Solution:

Our solution utilized state-of-the-art NLP models, including the Universal Sentence Encoder (USE) and BERT, to generate text embeddings for both user queries and document contents. These embeddings were indexed in OpenSearch/Elasticsearch running on AWS EC2 instances, enabling efficient retrieval of near-matching documents based on cosine similarity between query vectors and document embeddings. To further enhance user experience, we implemented techniques to extract relevant sections from retrieved documents, allowing users to quickly access the information they need.

Summary:

The successful implementation of our semantic search engine represents a significant advancement in information retrieval technology, providing businesses with a powerful tool to navigate and extract insights from vast document repositories with ease. By leveraging advanced NLP techniques and real-time indexing capabilities, we have achieved fetch times on the order of milliseconds, enabling users to access relevant information quickly and efficiently. This project underscores our commitment to innovation and our ability to deliver impactful solutions that drive efficiency and productivity for our clients.