Large language models (LLMs) have revolutionized the field of natural language processing (NLP) in recent years. They are capable of generating fluent and coherent texts on a variety of topics and domains, as well as answering complex questions and performing various tasks. LLMs are often associated with conversational applications, such as chatbots, voice assistants, and social media bots, where the goal is to interact with human users in natural language. However, there is another way to use LLMs that is less explored but equally promising: batch retrieval augmented applications (BRAG).
BRAG is a term coined to describe a class of applications that use LLMs to process large batches of text data and retrieve relevant information or insights. Unlike conversational applications, where the input and output are usually short and sequential, BRAG applications deal with longer and more diverse texts, such as documents, emails, reviews, reports, articles, and so on. The output of BRAG applications can be derived fields, tables, summaries, keywords, categories, sentiments, recommendations, or any other form of structured or unstructured data that can help the user, or the business achieve their goals.
BRAG applications and their ability to automate many tedious and time-consuming tasks involving large amounts of text data open different doors than traditional RAG applications. Here are some specific applications:
However, BRAG applications like the more traditional RAG applications, are not flawless and still often require a human in the loop to ensure quality and accuracy. LLMs are trained on large and diverse corpora of text data, which may contain errors, biases, or inconsistencies that can affect their performance and reliability. Moreover, LLMs may not be able to capture the nuances and contexts of specific domains or tasks, and may generate outputs that are irrelevant, inappropriate, or misleading. Therefore, it is always important to have a human expert or supervisor who can monitor, review, and correct the outputs of BRAG applications, and provide feedback or guidance to improve them. Human-in-the-loop systems can also help to increase trust and transparency in the use of LLMs and ensure that they adhere to ethical and legal standards.
At Spyglass we recently had the opportunity to apply BRAG for two customer use-cases. The first involved a university admissions department, struggling with the labor-intensive process of personalizing acceptance letters. Each student's acceptance required multiple drafts, leading to an extensive number of manual rewrites and slowed productivity.
The second challenge was presented by a medical device manufacturer. The company needed to review lengthy patient medical records, sometimes exceeding hundreds of pages, to determine candidate suitability for their devices. This process was time-consuming for medical personnel and delayed patient care.
To address these challenges, we implemented two BRAG applications. For the university admissions department, we developed a batch processing application that generated multiple personalized acceptance text samples for thousands of students. This application allowed admissions personnel to select the final text from the multiple generated options, streamlining the entire process.
For the medical device manufacturer, we created a BRAG application capable of processing extensive medical records. This application identified and summarized pertinent information, including page numbers where the information was found, significantly reducing the time required for review by medical staff.
The implementation of the BRAG application for the university resulted in quicker, more efficient production of personalized acceptance letters, freeing up admissions personnel to focus on other tasks.
For the medical device manufacturer, the BRAG application provided concise summaries of medical records, enabling medical staff to make faster decisions about patient care. This led to accelerated patient care and an increased number of patients processed daily, enhancing overall efficiency and effectiveness.
In the past we have built these applications using an Azure architecture with Data Factory, Azure Functions running in an App Service Plan and Azure SQL or Cosmos DB, API Manger, plus the supporting AI services. We typically use a web application as a way for the user to review the batch output. However recently we are moving to an architecture that leverages Microsoft Fabric as a platform.
Microsoft Fabric provides an excellent platform for batch-based applications such as BRAG. We use a Lakehouse to store the unstructured, or structured data to be processed. We use Data Factory as an orchestration layer to process the source materials through a medallion architecture (Raw, Cracked, Chunked), and then inserted into the Vector DB and AI Search Index. This is done using Fabric Notebooks.
We then use Fabric Notebooks running Lang Chain to run the prompt or prompts against the indexed materials and then persist the results back to the Lakehouse. With the results persisted back to the Lakehouse, we no longer need a web application; we can leverage Power BI as a presentation tier or even Power Apps using virtual tables.
This architecture allows us to minimize our Azure footprint and reduce the overall solution complexity.
If you're interested in learning how BRAG applications can be tailored to streamline your business processes, contact us today.