Exploring Agent-Based LLM Systems

This article explores the development and implementation of a Retrieval-Augmented Generation (RAG) system utilizing Large Language Model (LLM) agents to assist users in generating queries within Indeed’s proprietary big data environment, which employs the Imhotep Query Language (IQL). The system aims to reduce the complexity and barriers associated with IQL, making it more accessible to non-technical users familiar with SQL. By integrating a sequence of specialized agents — Thinker, Refiner, and Decider — each with distinct strategies, the system facilitates the query generation process, leveraging tools for data retrieval, syntax validation, and historical query analysis. The study evaluates the system’s performance, revealing that multiple agents improve query accuracy and user confidence, while the addition of IQL syntax prompts and the transition to more advanced models like GPT-4 enhance reasoning and output quality. Future work includes larger-scale user testing, the creation of an automated evaluation framework, and the refinement of compression methods to further improve the system’s efficiency and adaptability.

Introduction

The job marketplace platform, Indeed, uses a homegrown & open source storage system for big data and subsequent query language called IQL. This stands for Imhotep query language which leverages inverted indexes and is driven by a data compression technique built around time series data. Today, data at Indeed exists at a superhuman scale. IQL users find there is too much toil, cost, and…