Posts

Retrieval-augmented generation (RAG) is the dominant pattern for building LLM applications over private data. You retrieve relevant documents, pass them as context to an LLM, and get a grounded answer. But which LLM should you use? Does the dataset matter? And how do you even measure whether the answers are good?

To answer these questions, I ran 25,600 evaluations: 5 LLMs × 6 datasets × 2 retrieval conditions × 3 random seeds, with every response scored by 10 metrics — automated text similarity, a 3-judge panel, and RAGAS framework scores. The headline finding: automated metrics and the LLM judges disagree on which models are best.

There’s a growing gap in the AI agent conversation. Everyone’s talking about agents — autonomous systems that can reason, plan, and act. But most demos stop at “the agent wrote a nice email” or “it summarized a document.” The real challenge starts when you need an agent to interact with production backend systems, handle authentication, deal with partial failures, and return structured results that downstream systems can consume.

Over the past several months, I’ve been building exactly this: tool-equipped LLM agents operating in the payments domain. These agents can query transaction systems, look up order histories, check workflow statuses, retrieve standard operating procedures, and interact with ticketing systems — all through structured tool interfaces that the model invokes autonomously.

Here’s what I’ve learned.

Running and benchmarking python is easier if you have a standard place to run and share them. Kaggle fits this profile perfectly. It is a hosted service for your code which can be run and shared online.

Developing in Cloud

https://www.kaggle.com/tusharm567

As an added advantage, I get direct access to loads of public datasets to play and sharpen my skills on.

Benchmarking Code

from math import *
%timeit tan(atan(exp(log(sqrt(1*1)))))

This kind of stuff is helpful while learning and testing out algorithms and/or comparing different approaches to a solution based on compute.

After a long delay I finally mustered up the courage to build the query ranking module. Some scary stuff. Here are the problems I’ve been facing whenever I start building this.

• The ranking requires the table’s Foreign Key structure. Once the query generation is done, the recursive calls along with Javascript’s callbacks is a nightmare.
• Callbacks are being received even after sending the output. Some relations between the tables are discovered even after ranking. This is troubling.
• I tried to deal with promises but combined with the already scary ranking algorithm and multi-module structure this will just take more time than I have.

So I am switching to a new approach of this as suggested by a colleague.

Before I start building up the server for the application, I wanted to pen down the structure for the app. The structure is represented below.

Abbreviations:

• WSA: Web Speech API (Speech to Text)
• NLP: Natural Language Processing Tool (Syntaxnet or Google Cloud NLP API)
• DIS: Database Indexed Search
• DDI: Database Dynamic Indexing
• DES: Database Exhaustive Search
• NES: NoSQL Database Elastic Search
• SFT: SQL Database Full Text Search