Towards a harness that can do anything

I kind of have a different idea of agents. I totally believe in a deterministic scaffold but I really think that an agent should be as deterministic as possible - the more code, the better.

Think of a typical loop we may ask of Claude Code today (assume we are not using TDD): run some test suite with fail fast mode, diagnose if the failure is due to recent feature changes (pass reference to backend/frontend, github issues, PRD,...). Ask CC to decide if test failed due to feature change and then update the test. Perhaps ask CC to use sub-agent to investigate and fix (if deemed so). Commit each fix, move on to next.

I know, this has so many ways to make blunder but I am talking about the agent here, not our error-prone test maintenance. What if we had an agent that had context of your codebase, deterministically ran test suite, linter, hooks, etc. The "English" prompt would become a code loop with the LLM only brought in to decide if a test has failed because of feature change. Also, we can extract git log, JIRA and what not.

Each tool here is real code. Executable code that calls others and only prompts when they meet edge cases. Edge cases are defined but we can now accelerate the maintenance of these tools using agents themselves. But the system is built on "programs that do one thing and do it well" and then reach out to an LLM for its specific edge case. The agent is how these executables work with each other.

This post is chock-full of soft ideas. They make no meaningful steps toward "a harness that can do anything". The suggestion is to replace a small node application with specific tooling, with a vm "to give it more capabilities". This is what Agent sandboxes are, already. Making the sandbox the harness, doesn't achieve a concrete goal.

I use AI all the time

But

I don't want the AI to summarize my email for me.

I don't want the AI to summarize my calendar for me.

I don't want the AI to summarize my Zoom call for me.

Thanks

“That can do anything” gives me Zombo.com vibes. I would be more interested if it seemed to do particular things well.

Love the Unix philosophy and the buffs mentioned of the Linux FS. Lean, transparent, and auditability-first is exactly the direction harness' should continue in.

Something I am convinced of though, there probably isn't a single `best` harness for all tasks. Different workloads will likely perform better with certain combinations of model + harness, especially when we are talking about token budgeting and cost tracking.

Ambiance feels like a great base “kernel” to build those variants on top of, rather than the one true harness.

What's with this "harness" word people have been trying to adopt lately? Are we all going rock climbing?

What has been the most helpful when developing harnesses:

> When in doubt, simplify. Remove, trim and minimize. Reproduce issues in as small cases as possible, understand the full design completely, there is no shortcuts for this.

How much do the labs post-train on the harness inputs & outputs? That's a critical piece to understand if a "generic" harness is at all possible

I really like this idea and the way you mapped the concepts to unix primitives. Indeed llms are already "unix native". I've been experimenting with similar event driven workflows using k8s primitives but that's one level up the stack. This makes a whole lot of sense to me in terms of organizing a shared mental model. Will definitely check it out. Thanks for the good work.

I think domain specific harnesses are already surpassing generic harnesses. I also think software development is its own domain.

My harness is a Claude Code plugin with its own brainstorming, adr, and planning skills with associated review and interview skills. Behavioral testing related to acceptance criteria is built in. Everything in my harness is gated to prevent ratholes.

I recently inflated a docker container to execute a set of work with Claude in unsafe mode and immediately saw problems with everything it was doing…and then I realized I had not installed my harness.

Running Claude without an engineering harness is like driving a car without brakes or a steering wheel.

Mapping agent concepts to the Unix environment sounds like a great idea in general, but I get off the train at adopting the FHS, which is an ancient relic that has no place in a green-field system. I don't know exactly what shape it should take, but something along the lines of Nix seems more appropriate. Or maybe Plan 9.

I disagree with the idea that file is a good metaphor for LLM. Files have seek and byte streams, which is just an unneeded abstraction for LLM. The LLM doesn't need to seek or jump to the middle of a file, if you store and organize your data properly.

Why force the LLM to use files over vector database or key-value stores, just because it's a design principal for UNIX (which is designed for human users, not LLMs.)

>I've been thinking about how to free LLMs from the chat pane for a few years now.

LLM's are language models, you can absolutely control them with bash scripts and deterministic code, there are plenty of frameworks that already do that, and a great engineer will use them, but LLMS are at their most powerful when a user can give the agent an input and the model can run its ReAct loop. Wanting to free an LLM from a chat pane is like wanting to free email from the thread model, or closer, removing the chat window to DM a friend or colleague.

>what can we learn from the before-fore times, when people used to actually write code?

Treat an agent like a human writing code. Give them the best context, give them the best tools. This is why harnesses are overly complicated, because they need to guide the model through the context and tools it has available in a way that is efficient. A good harness is not incompatible with the Unix philosphy, it can do one thing well (interfacing with LLMs and giving them access to filesystems and compute), it will heavily use bash, stringing commands together with the cli tools that it knows (it's context) that it has, and and LLM will naturally handle text streams because that is what it does best.

>Everything is a File

If you want things to be deterministic why resort to plaintext? Wouldn't we want as much as possible to be typed? A computer can parse json which is what you want if you are trying to make your harness as deterministic as possible.

>It watches our FS for changes with cursors on textfiles,

Wow. What is your monthly token bill? I don't know how that would use less tokens than a 30 minute heartbeat, which as you mention will already use a lot of tokens. Why not have it notify your agent after a certain amount of files have been changed, or certain files you deem important?

It seems like this user works at a 12 week programmer retreat and seems to post their cohort's blog posts about the projects they work on.

I was hoping this was a climbing harness that could service other trades too

Why are we not just using Claude Code or Codex on our machine and using this thing? Real question...

Just one more harness bro, one more harness I swear one more harness and we will have solved AGI bro

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]