Show HN: Needle: We Distilled Gemini Tool Calling into a 26M Model

Do you have any examples or data on the discriminatory power of the model for tool use?

The examples are things like "What is the weather in San Francisco", where you are only passed a tool like

  tools='[{"name":"get_weather","parameters":{"location":"string"}}]',

I had a thing[1] over 10 years ago that could handle this kind of problem using SPARQL and knowledge graphs.

My question is how effective is it at handling ambiguity.

Can I send it something like a text message "lets catch up at coffee tomorrow 10:00" and a command like "save this" and have it choose a "add appointment" action from hundreds (or even tens) of possible tools?

[1] https://github.com/nlothian/Acuitra/wiki/About

Hmm.. this might make it feasible to build something like a command line program where you can optionally just specify the arguments in natural language. Although I know people will object to including an extra 14 MB and the computation for "parsing" and it could be pretty bad if everyone started doing that.

But it's really interesting to me that that may be possible now. You can include a fine-tuned model that understands how to use your program.

E.g. `> toolcli what can you do` runs `toolcli --help summary`, `toolcli add tom to teamfutz group` = `toolcli --gadd teamfutz tom`

Suggestion: publish a live demo of the "needle playground". It's small enough that it should be pretty cheap to run this on a little VPS somewhere!

Are you worried about Google's response to this? Google reportedly reacts to distillation attempts "with real-time proactive defenses that can degrade student model performance". So if they detected you, they could have intentionally fed you a dumber but plausible variant of Gemini: https://cloud.google.com/blog/topics/threat-intelligence/dis...

But also, this model is small and just focusing on the tool use. In terms of token usage, you're probably not anywhere near the people that are trying to distill the entire model.

>Experiments at Cactus showed that MLPs can be completely dropped from transformer networks, as long as the model relies on external knowledge source.

Heh, what a coincidence, just today one of my students presented research results which also confirmed this. He removed MLP from Qwen and the model still could do transformation tasks on input but lost knowledge.

This is neat, and matches an observation I saw with early Claude Code usage:

Sonnet would often call tools quickly to gather more context, whereas Opus would spend more time reasoning and trying to solve a problem with the context it had.

This led to lots of duplicated functions and slower development, though the new models (GPT-5.5 and Opus 4.6) seem to suffer from this less.

My takeaway was that “dumber” (i.e. smaller) models might be better as an agentic harness, or at least feasibly cheaper/faster to run for a large swath of problems.

I haven’t found Gemini to be particularly good at long horizon tool calling though. It might be interesting to distill traces from real Codex or Claude code sessions, where there’s long chains of tool calls between each user query.

Personally, I’d love a slightly larger model that runs easily on an e.g. 32GB M2 MBP, but with tool calling RL as the primary focus.

Some of the open weight models are getting close (Kimi, Qwen), but the quantization required to fit them on smaller machines seems to drop performance substantially.

That M versus B is way too subtle. 0.026B is my suggestion

Awesome! I just tried to set an alarm and add some groceries to the shopping list, and it outperformed Siri.

Lovely to see the push for tiny models.

I have been building for small (20B or less) models for quite a while. Highly focused/constrained agents, many of them running together in some kind of task orchestration mode to achieve what feels like one "agent".

I build (privacy first) desktop apps this way and I want to get into mobile apps with similar ideas but tiny models.

I'm so excited for this, nice work!

Gemma4 edge models were promised to be great for agentic use, but have been really disappointing in all my tests. They fail at the most basic tool use scenarios.

Have you run any tool-use benchmarks for Needle, or do you plan to? Would be great if you could add results to the repo if so.

Dumb questions, from someone not in the field...

What is a distilled model?

Why doesn't Google do this (to make their models smaller)?

Seems like you could make a competitor to Gemini?

A lot of agent workflows really are just tool selection + argument extraction + structured output. How does this behave once workflows become multi-step and state starts accumulating across calls?

How could you use this for composability? I.e. chaining together multiple tools. For example web_search → summarize_url → send_email

Looks like you need to open up access to https://huggingface.co/Cactus-Compute/datasets/needle-tokeni... - I get this error when trying to run the steps in your README:

> Repository Not Found for url: http s://huggingface.co/api/datasets/Cactus-Compute/needle-tokenizer/revision/main.

Sounds interesting.

Got a bunch of errors trying to run it on CPU though. Very likely connected to me running this in a container (unpriv LXC), but figured for 26M CPU would suffice.

https://pastebin.com/PYZJKTNk

This is pretty much exactly what I want for Home Assistant. I yell out, "Computer! Lights!" and it toggles the lamp in the room on or off. (I mean I can do that now, I think, but probably with a much larger model.)

I haven't played with it yet, but does it ever return anything other than a tool call? What are the failure modes? What if it doesn't understand the request? Does it ever say it can't find a tool? Does it get confused if there are two similar (but different) tools? Can it chain tools together (e.g. one tool to look up and address and another to get directions to the address)?

I mean, I plan on downloading the model later tonight and finding out for myself, but since I'm stuck at work right now, I figured I'd ask anyway...

Can it summarize text it fetches?

Come to think of it, this could be a nice model to have as the first pass in a more complex agent system where Needle hands of the results of a tool call to a larger model.

I will defiantly play around with this!

From all the models that do toolcalls the only thing I am confused is why did you pick the worst? Or maybe they are only bad in agentic work it fine for one shot toolcalls?

Can this be a Siri-like core? Set me a timer, tell me what’s the weather, etc. Here is transcribed text and available list of tools for the model to call, and voice the output.

I don't really understand what this is for... there is a lot of ML-researcher talk on the GH page about the model architecture, but how should I use it?

Is it a replacement for Kimi 2.7, Claude Haiku, Gemini Flash 3.1 lite, a conversational LLM for the situations where it's mostly tool-calling like coding and conversational AI?

This would be amazing for home assistant.

I find this stuff super fascinating and been thinking about it myself. Maybe one could bootstrap tiny models on a rather 'pure' procedural data set. Neglecting [0] of course...

[0]: http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Is the idea here to add function calling to models that don't have it, or even improve function calling (qwen quirks)?

No FFN is blowing my mind. This is pretty much "Attention Is ACTUALLY All You Need". Reminds me of BERT Q&A which would return indices into the input context, but even that had a FFN. Really exciting work.

Nice catch. Using agent for simple tasks is inefficient and wasteful, Needle really resolves this. Looking forward to future upgrades!

Does the model have capacity for in context learning ?, if we give it examples of patterns can it follow them ?.

Why pick Gemini? It's probably the worst tool calling model of the major labs.

Can this be converted to onnx or otherwise be used in a browser?

Query: set a timer for 1 hour

Result: [{"name":"set_timer","arguments":{"time_human":"1 hour"}}]

Query: in 1 hour set a timer for 1 hour

Result: [{"name":"set_timer","arguments":{"time_human":"1 hour"}}]

I'd expect either a chain load or just a 2 hour timer. Further attempts humorously give two separate 1-hour-timers.

This is some excellent work Henry! Very excited to try it out.

This is very cool I'm going to try to carve out some time to try building this into my MOO system ( https://codeberg.org/timbran/moor / https://timbran.org/moor.html ) as alternative command parser front end.

This is really cool. Any plans to release the dataset?

hey nice work, is it possible to release the datasets?

I assume this would only be useful as the second stage after a model like Whisper, as it can't understand speech where you'd want it, like on a phone or small device?

What is the use case for this?

I source old, defective high-end radios with timeless designs from brands like Grundig or Braun, and replace the original hardware with a Raspberry Pi while using the original audio parts to build custom smart speakers. Reliable hotword detection and voice command recognition have been a persistent challenge over the years, but whisper and other small models have helped enormously. At the moment I have ollama running on my server with qwen 9b which works fine but a 26M that could be deployed on the pi itself would be amazing.

[flagged]

[dead]

[flagged]

[flagged]

[flagged]

[flagged]

[dead]

[dead]

[dead]

[flagged]

[flagged]

FYI, distilling Gemini is explicitly against the ToS:

"You may not use the Services to develop models that compete with the Services (e.g., Gemini API or Google AI Studio). You also may not attempt to reverse engineer, extract or replicate any component of the Services, including the underlying data or models (e.g., parameter weights)."