{
    "componentChunkName": "component---src-templates-blog-template-js",
    "path": "/blog/optimizing-tool-selection-in-llm-workflows-part-2-a-dspy-pytorch-benchmark",
    "result": {"data":{"markdownRemark":{"html":"<p><em>All code is available on github <a href=\"https://github.com/viksit/differentiable-programming/blob/main/tool-call-eval.py\" target=\"_blank\" rel=\"noopener noreferrer\">here</a>.</em></p>\n<p class=\"article-image article-image--lead\"><a href=\"/images/substack/f9829ff93f52fc20eb8a0045.png\"><img src=\"/images/substack/f9829ff93f52fc20eb8a0045.png\" alt=\"\"></a></p>\n<hr>\n<p><a href=\"/blog/optimizing-tool-selection-for-llm-workflows-differentiable-programming-with-pytorch-and-dspy/\">In a previous post</a>, we explored how a lightweight neural controller — a tiny PyTorch module — could replace prompt-based routing in LLM workflows. That post struck a chord. <a href=\"https://news.ycombinator.com/item?id=44475453\" target=\"_blank\" rel=\"noopener noreferrer\">It was on the front page of HN</a>. Readers chimed in with sharp questions. <em>Could this scale? Was it accurate? What about argument generation? How does it hold up against GPT in the wild?</em></p>\n<p>This week, I tried to answer those questions with an eval. I took the most obvious benchmark: customer support tickets, and asked a very specific question: can a local controller match GPT’s performance in choosing the right path through a workflow?</p>\n<p>And perhaps more importantly: when it can, why should we care?</p>\n<h2>A Question of Responsibility</h2>\n<p>There’s a pattern I see again and again in modern agentic workflows. It looks simple on the surface:</p>\n<ol>\n<li>A user submits a query.</li>\n<li>An LLM decides which tool to call.</li>\n<li>The tool executes.</li>\n<li>Another LLM interprets the output and produces a reply.</li>\n</ol>\n<p>This chaining of LLM calls feels elegant because you hand over the entire workflow to a language model and let it orchestrate the steps. It’s fast to prototype, flexible to adapt, and easy to debug — but up to a point.</p>\n<p>Each decision point — “<em>Should we refund the customer?</em>”, “<em>Should we escalate this?</em>” — becomes a full GPT call. And with each call comes more tokens, more cost, more context to reintroduce. You find yourself burning inference time on problems you’ve already solved.</p>\n<p><strong>Moreover, the LLM isn’t even learning from its past choices</strong>. It makes the same inference again and again, in a stateless and expensive manner</p>\n<h2>What We Set Out to Measure</h2>\n<p>This post is about replacing that first decision step — the tool selection — with a learned function.</p>\n<p>We wanted to answer a narrow but important question:</p>\n<blockquote>\n<p>Can a local controller, trained on a few thousand examples, match GPT’s ability to decide whether a customer should get a refund?</p>\n</blockquote>\n<p>To test this, we created a dataset of 1000 support tickets, synthetically generated and labeled using GPT-4o. These tickets were phrased to mimic real-world conditions: short, noisy, sometimes vague. They asked about late deliveries, missing items, unresponsive drivers — the typical long tail of customer frustration.</p>\n<p>We split this dataset into 800 training examples and 200 test examples (after augmentation). Then we trained a simple RNN to act as a binary classifier..</p>\n<h2>Building the Controller</h2>\n<p>The controller was deliberately minimal. A small embedding layer, a GRU, and a linear output — that’s it. No pretraining, no tricks, no fancy encoders. (Note: this could be as complex a model as we like, from simply using OpenAI embeddings to using a fully fine tuned ToolCall model).</p>\n<p>This model receives raw text, tokenized into word-level indices using a vocabulary built on-the-fly, and learns to output a softmax over two choices: REFUND or NO_REFUND.</p>\n<p>Training took less than 10 seconds on a Google Colab T4 instance. Accuracy on the held-out set was perfect — not because the model was overfit, but because the classification task itself was simple. The head queries (late order, missing item) dominate the distribution.</p>\n<p>And here’s a simple sanity check.</p>\n<p class=\"article-image\"><a href=\"/images/substack/e6cfd3b17d7f3d12f2773230.png\"><img src=\"/images/substack/e6cfd3b17d7f3d12f2773230.png\" alt=\"\"></a></p>\n<h2>Routing with DSPy</h2>\n<p>Next, we built two agents using <a href=\"https://github.com/stanfordnlp/dspy\" target=\"_blank\" rel=\"noopener noreferrer\">DSPy</a>, a structured framework for composing LLM modules into pipelines:</p>\n<ul>\n<li><strong>Agent A</strong> used GPT-4o for both classification and reply.</li>\n<li><strong>Agent B</strong> used the trained RNN for classification, and GPT-4o for the reply.</li>\n</ul>\n<p>The only difference being: who decides.</p>\n<p class=\"article-image\"><a href=\"/images/substack/bd2fc7fa5e8f889325abc9c9.png\"><img src=\"/images/substack/bd2fc7fa5e8f889325abc9c9.png\" alt=\"\"></a></p>\n<p class=\"article-image\"><a href=\"/images/substack/28157dfef08f244e1e4624b5.png\"><img src=\"/images/substack/28157dfef08f244e1e4624b5.png\" alt=\"\"></a></p>\n<p>This was a clean separation of control logic from generation. The reply prompt was exactly the same in both agents. Only the routing mechanism varied.</p>\n<h2>Evaluation</h2>\n<p>We evaluated both agents using DSPy’s built-in Evaluate block and captured the actual cost via dspy.LM.history, which records token usage and pricing metadata for each LLM call.</p>\n<p class=\"article-image\"><a href=\"/images/substack/2f0ad3c86c87c8d609a2888f.png\"><img src=\"/images/substack/2f0ad3c86c87c8d609a2888f.png\" alt=\"\"></a></p>\n<p>The RNN controller matched GPT in accuracy. And because it handled classification locally, it cut API usage in half. That translates to a 40% reduction in cost, with a negligible increase in end-to-end latency.</p>\n<p>And that latency could be reduced further. The current implementation evaluates the RNN serially, without batching, and runs on CPU by default. Even with those handicaps, the controller is competitive.</p>\n<h2><strong>What’s the Takeaway?</strong></h2>\n<p>This experiment was deliberately simple. It was meant to do an eval of one idea:</p>\n<blockquote>\n<p>What happens when we remove tool selection from the prompt, and treat it as a learnable layer?</p>\n</blockquote>\n<p>The answer: nothing breaks. In fact, everything gets easier.</p>\n<ul>\n<li>The system becomes more modular.</li>\n<li>The model becomes more inspectable.</li>\n<li>The workflow becomes cheaper to run.</li>\n</ul>\n<p>More importantly, it hints at a future where control flow lives outside the LLM. Where we build agents that act more like programs — with structure, abstraction, and reusable parts.</p>\n<h2>What’s next?</h2>\n<p>The next frontier is argument learning — teaching controllers not just which tool to call, <strong>but how to call it</strong>.</p>\n<p>If a refund is due, how much? If an escalation is needed, to whom?</p>\n<p>In future posts, I’m going to explore the synthetic-history trick: passing in fake traces so that the LLM remains in-distribution even when routing is offloaded. And comparing this controller to small finetuned language models — the kind you might run locally in production. When I find some cheap GPUs to test on.</p>","excerpt":"All code is available on github here.  In a previous post, we explored how a lightweight neural controller — a tiny PyTorch module — could replace prompt-based…","frontmatter":{"date":"July 23, 2025","rawDate":"2025-07-23","substack_post_date":"2025-07-23T20:44:27.317Z","slug":"/blog/optimizing-tool-selection-in-llm-workflows-part-2-a-dspy-pytorch-benchmark","title":"Optimizing Tool Selection in LLM Workflows (Part 2): A DSPy + PyTorch Benchmark","ispage":false,"ispublished":true,"source_url":"https://viksit.substack.com/p/evaluation-tool-selection-in-llms","description":"Last time, we showed how a custom PyTorch controller could replace prompt-based routing inside agent workflows. Now we put that to the test."}}},"pageContext":{"slug":"/blog/optimizing-tool-selection-in-llm-workflows-part-2-a-dspy-pytorch-benchmark","next":{"frontmatter":{"slug":"/blog/behavioral-optimization-for-multi-step-agents-metrics-and-rewards-in-dspy","title":"Behavioral optimization for multi-step agents: Metrics and rewards in DSPy","date":"2025-08-06","ispublished":true,"ispage":false},"slug":"/blog/behavioral-optimization-for-multi-step-agents-metrics-and-rewards-in-dspy/","topic":"agents","kind":"technical article"},"previous":{"frontmatter":{"slug":"/blog/differentiable-programming-for-learnable-graphs-optimizing-agent-workflows-with-dspy","title":"Differentiable Programming for Learnable Graphs: Optimizing Agent Workflows with DSPy","date":"2025-07-13","ispublished":true,"ispage":false},"slug":"/blog/differentiable-programming-for-learnable-graphs-optimizing-agent-workflows-with-dspy/","topic":"agents","kind":"technical article"}}},
    "staticQueryHashes": ["4202924991"]}