Using evals to build robust chatbot assistants

I’ll be demonstrating a project I am working on where we are using evals to build robust LLM pipelines. Evals help us be more confident shipping LLM based apps to production.

I am building a chatbot assistant that is functioning as a Natural Language to SQL agent, allowing users to interact with data in the database, and to get insights from the companies data. This is an actual live project for a business in the UAE

The demo will be focusing on how we built evaluation sets to help write deterministic tests for the agent, to give us more confidence in its performance. We will discuss

• how we break down the potential query space of the assistant into defined categories using dimensions and error analysis. This helps structure the set of queries we want our bot to work for, as oppose to having an infinite query space which is hard to design good products around. This gives us structure for designing a well performing assistant
• we will show how we have written automated evaluation tests, where we pass queries to our bot, check the results it gets from the db, and we validate that against expected data. This is basically writing deterministic unit tests for our non-deterministic agent pipeline