Drawgent: Coding agent on a live Excalidraw canvas

I think many folks are underutilizing or underestimate the power of HTML.

There's been a few of these things on the front page lately. Including false "whiteboards" that are DSL abstractions over things like React Flow...

I've done quite a bit with Excalidraw MCP and the models had to deal with a lot of JSON data and estimating/computing with bounding box and pixel point numbers.

But with HTML, the agents get the natural semantics/styling capabilities and basically zero abstraction. The recent models can draw exceptionally well any form of logical/conceptual/flow diagram.

Excalidraw offers their own open source first party MCP endpoint[0] and server[1]:

[0]https://mcp.excalidraw.com

[1]https://github.com/excalidraw/excalidraw-mcp

I've had a fairly thorough exploration of how I can give my agent a whiteboard so we can work on architectures together. To my surprise, current solutions (including Excalidraw) were not good enough and didn't deliver what I wanted. I ended up finding Mermaid to be the most agent-friendly medium and coded an Obsidian plugin for it. It works okay; we can work on the same doc while I bring my own AI agent, and we can brainstorm together. https://community.obsidian.md/plugins/mermaid-relay

YMMV, but the value I get from producing a diagram is derived from the thinking. Thinking about what I'm trying to draw leads to understanding about where my/my team's knowledge is poorer, what assumptions we're making, etc.

This is crazy. My claude chess stuff is (edit: was) currently near yours on the front page and noticed your post. I have a project very close than yours that I hesitated to share. From a quick glance, we went for a similar approach. I just open sourced it so that you can compare implementation notes. https://github.com/brumar/whiteboard-agents . It's not thoroughly tested but can be interesting to check.

Posted this somewhere else aswell but I've had good success with https://whiteboard-mcp.com - it seems to work better than other free alternatives for things like architecture diagram creation

I have a personal experience that there is a lot of attention be gained/exploited with tools such as Excalidraw because people still don't understand that this is possible to automate, and people with an LLM-negative bias that are actively against generated content will still give attention to such tools because it looks hand-drawn.

For example /r/art banned AI-generated content without a process due to this anti-automation bias, however you can now generate the process by using MCP-server tool calls into the software.

So there is a big attention market as long as you can keep up with things that seemingly looks like they had human effort.

What frustrates me about many projects is it's never been easier to just make a quick video or image or gif and put it in the README to show what the value is. This was at least somewhat complex before AI but now you can just paste an image and have the AI put it in your README.

I guess the frustrating bit is that the readme is 100s of lines long. AI is already saving you so much time on a project like this but you still make me clone and spin it up if i want to see it for real.

I get that open source is a gift and you’re not obliged to do anything I say but please consider taking the extra few minutes on your next project.

https://likec4.dev/ It's a bit different usecase, but llms are proficient with it

I’ve been trying to find the best practice to connect a person’s existing ChatGPT/Codex / Claude accounts so they can be used in an app for the agent in the app.

Any advice? Everything seems clunky, even the best MCP efforts

video https://lnkd.in/p/ez4EZede

I am working on a similar thing for jsoncanvas (.canvas files in Obsidian). My takeaway is that agents are heavily trained on specific formats (e.g. svg), and are conversely pretty bad at niche formats. My strategy was to create a native renderer [1] so that at least they can rapidly iterate "visually" by reading svg and png renders instead of trying to one-shot.

I have yet to see an AI generated visual diagram that doesn't feel like slop though... I'd be very interested in examples of actually-good diagrams workflows if anyone has seen them.

[0] https://jsoncanvas.org/

[1] https://github.com/dtkav/canvas-native

tldraw has a lot of programmatic access I have not compared it to excalidraw but there is still no equivalence between the model capabilities to build amazingly detailed infographics and diagraming tool use which is a gap I hope will narrow