Small Models Have Arrived

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off.

There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was before "thinking" models, and yet using that library I was able to "guide" the model in the required "prompt / instruct" context such that it was working towards completion, and I saw the first things like we see now in the thinking traces "oh, test x doesn't pass because blah, I need to..." and so on.

Anyway, the revelation was "even if the models never improve, I'll have years of fun finding out all the ways I can use these things". And, obviously, the models improved a lot since then. But I think that revelation can still be applied, as a sort of "truism". We have, right now, access to things that 10-20 years ago would be considered magic. We are still finding ways of cobbling together systems with glue, duct tape and prayers and find new things they can do.

I think the "good-enough" stage has come not just for API models (cheap, fast, etc) but for local as well. Even if slower, even if clunkier, but they are good enough for a set of ever increasing tasks, and what's more it's incredibly fun to work with them.

> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?"

What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start.

Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do the hard part of understanding specific consumer needs and wants and pursuing that.

> Across his various startups, Peter has seen two kinds of work:

> 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of.

> 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts.

Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html

I'm curious about not only which of these roles models will fill, but also how they will empower us to be in the mode we prefer.

We mostly use specialized small models because larger are to expensive/slow and are prone to hallucinating quite a bit. Not sure how this is a surprise, seems more like a best practice.

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model.

There are many applications where world knowledge is unnecessary or even a negative, and in which only a small amount of language skill is necessary, and there we can expect small models more intelligently used to beat large ones naively used.

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

As someone who uses a lot of opus, I'm curious what changes to my workflow I need to make to make small models work. My workflow currently is for a large feature I'd explain the feature and how I envision it technically and run the grill me skill where it'd ask me questions that I may not have thought of, then it'll write a spec and execute from there. It spawns an orchestrator so supposedly its supposed to use haiku for mechanical things but im not confident that works.

How much of this can be replaced with small models, and what changes to my workflow do i need to make?

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

I’m kind of cautiously excited for the next five to ten years, with these AI chips becoming incredibly fast and RAM capacities ramping up its in the cards that we’ll have chips like today’s ATMEL microprocessors that fit on a single board computer and can run small models locally, then all our gizmos can have local AI and I can have a truly intelligent home. Of course there will be a huge push to put all of it in the cloud but maybe we have a chance to take this technology home for good as it’s hard to imagine people will submit to this kind of surveillance required for AI home automation 24/7 (then again I might be wrong). Exciting times.

A dream of mine is to be able to host a LLM-powered video game that I can host on a home server running a decent mid-range GPU like the RTX 5060, and the LLM is fast and intelligent enough to make for a fun game experience for a few dozen concurrent players. People can ask for features and they just get made and added to the game on the fly for the lobby to enjoy. The hosting costs would be manageable enough that I don't have to charge anything for the game.

I think with one more year or so of small model progress, that might just be possible to accomplish.

Luna is definitely "good enough" for so, so many use cases. And that price point...

Goes to show that open source = cheap, closed source = expensive is wrong-headed. Closed source is going to beat open source on price point due to scale (try buying a few GPUs to run your own model on...) and labs innovating across the full inference stack.

That leaves the value prop of open source as privacy / customizability.

Luna max is suitable for like 90% of the kinds of code changes I want to make. I only find myself actually reaching for a Sol or Fable tier model if the problem is very complex. If you're willing to build the guardrails and do some extra planning, Luna is very capable.

> There's obviously a lot we can optimize here, but if you're charging what the WSJ or The Economist charges, you'd better be delivering similar value.

Gosh, watching paint dry has been a better value than reading The Economist in the last 5 years or so.

That aside, I had good results with Luna. I'd be interested in hearing about a comparison that takes into consideration response time (not TPS), cost and performance of the popular models at different settings. That chart has some of that. For instance, is Luna Max a better value than Terra Medium?

Nobody is questioning what is meant by small? When you started mentioning frontier models that don't run locally, I just go TB;DR "Too big, didn't read"

I don’t care about the cost if the results are weak. I did a bunch of work with Sol only to find Opus 5 spot a bunch of bugs.

And it was correct. The results weren’t as good as they should have been for Sol. How am I going to trust Luna?

Regarding the Pareto frontier and related benchmarks, I have a hard time taking anything seriously that claims that Opus is anywhere near the intelligence of Fable. Are there any benchmarks that haven't just been benchmaxxed that more accurately represent actual usage?

"Small models" nowadays work like someone who has IQ 100+ while SOTA ones are like 150, "relatively".

Given sheer number of turns I can make with small models, I can do a lotta stufff

- cheaper, and faster

Harness makes differences: There have been many HN posts about how one made tiny models work better at certain tasks using harnesses.

These "small" models with right context, and guidance, they work wonders.

---

I've been saying Luna has been my go-to AI in previous comments and why Luna is still more compelling than GLM-5.3-flash.

- https://news.ycombinator.com/item?id=49450353#49452248

Running small models locally beats wrestling with API latencies and rate limits. The compute trade-off is 100% worth the privacy and DX gains.

Small is relative. I'm looking for models that I can with run around 100 MiB mark (RAM just for the weights) to demo what you can do with this little memory.

I know of SmolLM 2 which in Q4 is borderline regarding the size and rather dated. There is also TinyStories, which is also old and also focussed on children's stories.

Is there anything newer in this category? Or should I try to distill something down to this size?

I kind of agree with your assessment. Running models not just on local, but cheap and lightweight frameworks will drive the next phase.

Not trying to plug, but I do't know any other way. I wrote a piece couple of days ago on small models and memory usage on the edge devices- https://polign.com/blog-edge-agent-memory and https://news.ycombinator.com/item?id=49450816 closing on the same problem.

It's kinda fun that people get to experience what the 8-bit era was like!

IMO big models are not a product in and of themselves. Inference is just a new type of compute. I'm confident that in two or three years, every product will have inference capabilities integrated into the experience, and models will become less and less distinctive from one another.

What most products need from a model is a pretty short list: the ability to make tool calls well, accurate recall, and the ability to follow directions without wavering (whether or not those directions are baked into the weights or provided in a system prompt). That covers 95% of inference utility in products.

We're nearly there, and I believe these capabilities will fit on small models.

Because of this though, I predict hardware demand will stay high despite demand for "hosted" inference dropping. Unless there's some regulatory shenanigans that step in to say otherwise.

> the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts... ~95% of the work he does falls into bucket 2. It's hopping on calls. Nudging people. Blocking and tackling.

This is a good insight broadly!

The future is self-hosted models trained on your own reasoning traces autonomously, as you sleep, using QLoRA and whatever else.

The tokens per second speed measurement is highly inflated nowadays because most of the tokens went into thinking. I wonder if there is a more realistic measurement for "effective speed", which accounts for thinking efficiency.

100% agreed. Small, cheap, and hosted models. Luna (and open weight models and others) is ridiculously cheap @ $0.2/$1.2, easily accessible, and more than good enough for basic use cases (e.g. summarization, simple tool calling, etc.).

I love how "Small Models" apparently is "Model of unknown size but probably smaller than another model that we also don't know the size of".

An absolutely nothing post at #2 on the frontpage

Maybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local models" in the cloud providers like DigitalOcean, AWS, etc.

The cost difference is pretty dang nice. Going from roughly a dollar to $0.10 for the same kind of task makes it so that products that didn't make financial sense before become possible

The demand for fast, cheap, good enough models has always been borderline infinite, it’s the supply that’s going to take off.

I forked my Big Serious Harness™ that models construction projects into a harness for building a vibe coded family assistant. I couldn't figure out how to make the toy operate at toy prices until Luna. Now you can vibe code all the little apps you might want for your fam for like $5 and operate it day to day for a few cents.

> Peter runs multiple companies. Beyond Segment, he's raised $100m+ for Charm Industrial, and just recently closed a Series A for Revoy. He's incredibly organized and efficient with his time.

You can do this before an exit? Build and fundraise for multiple (3?) companies at the same time?

Watch reasoning tokens though. We tried a small reasoning model that burned ~2800 thinking tokens per call, 3x the cost of a cheaper non-reasoning one despite a better price sheet.

you both deserve a hot place in the HELL

I think we'll see more of this soon

replit is already leading the way with free luna usage

The "good enough" concept is interesting because of how systematically people over estimate it. So often, things that are lower quality but thought to be "good enough" turn out to be either not good enough or not worth it compared to just using the higher quality "thing".

I will believe that smaller models have hit that bar empirically when I see them in production. At the moment, even frontier models are stuck in most of the scenarios I am seeing for high value tasks at the "not good enough" gate - so small models are not even close to being on the scene there yet.

using small local models - with a little bit of extra work - for the first time over the past few days was _really_ illuminating and inspired similar thoughts about how far you can practically get with so little. column of zap emojis, mane ...

That's why the market has our solutions but economics.

I recently had some relevant experience: for a couple of months now I've been experimenting with on device models to summarize feeds in a Bluesky client I am developing. The feature extracts topic areas, categorizes posts, and creates a summary under each topic.

At first the results were hot garbage, and progress was slow. I hooked up the settings to download models from Hugging Face conveniently, so I could run experiments faster, and I massaged the prompts a bit. Last week this feature made a qualitative jump from science experiment to something I'd actually use.

The fact that all runs on the device means I've got no variable costs associated with adding this to what will be, at best, a pretty low revenue product. I've tested it on trailing edge devices like an M1 Mac and a Pixel 8, and performance is very tolerable.

The key is I'm not asking for open ended answers to open ended problems. When it proves to be useful it's not going to get less useful or more expensive.

There are vast domains of uses for LLM models with similar characteristics and likely similar results.

Nice!!

small models + a good application layer - are more than enough, good for routine business tasks.

the application Layer i.e having a good graph RAG & connecting it up together is the missing piece for most.

how effective are the small model?

I have trouble seeing the points of using less capable models.

I just want the smartest, best, and most capable models. It feels smaller models for speed and cost are just transitions towards better hardware allowing the very best model.

you both derve a special place in HELL

I like the analogy on ways to make small model useful.

not sure how this is really about model size.

I'm keenly interested in seeing super small (like 10's or 100's of mb) special purpose models starting to come into their own.

that might actually be transformational globally not just in rich countries.

100 t/s of crap is still crap.

Well said

somehow I cant understand how luna is a step up. to me it feel dumber than 4.1. slower too

[dead]

[flagged]

[flagged]

[flagged]

[dead]

[flagged]