Bombadil: Property-based testing for web UIs

I'm a huge fan of property based testing, I've built some runners before, and I think it can be great for UI things too so very happy to see this coming around more.

Something I couldn't see was how those examples actually work, there are no actions specified. Do they watch a user, default to randomly hitting the keyboard, neither and you need to specify some actions to take?

What about rerunning things?

Is there shrinking?

edit - a suggestion for examples, have a basic UI hosted on a static page which is broken in a way the test can find. Like a thing with a button that triggers a notification and doesn't actually have a limit of 5 notifications.

I especially like that it's a single executable according to the docs.

Recently evaluated other testing tools/frameworks and if you're not already running the npm-dependencyhell-shitshow for your projects, most tools will pull in at least 100 dependencies.

I might be old fashioned but that's just too much for my taste. I love single-use tools with limited scope like e.g. esbuild or now this.

Will give this a try, soon.

I'm doing propety-based test since years for frontend stuff. The hardest part is, that there is so much between the test inputs and the application under test, that I find 50% of the time problems with the frontend test frameworks/libs and not in our code.

Author here, happy to answer questions about Bombadil! :)

This looks genuinely awesome! I've been thinking about how to do good property based testing on UIs, and this elegantly solves that problem — I love the language they've designed here. It really feels like model checking or something.

Hey Oskar ~ great project and looks promising. I would be curious to hear what is still work-in-progress for Bombadil.

It's helpful to know what the tool maintainers see as upcoming or incomplete work. It also saves a consultant like me a lot of time to evaluate new tools for clients if I also know the limitations before diving in. Maybe a section in the manual for "What Bombadil can't do".

Great work!

Congrats to the team!

Is there a video showing someone spinning this up and finding a bug in a simple app?

A broken counter app maybe?

The major breakthrough here is that they managed to write a project in Rust and not mention it on the headline!

Jokes aside, great project and documentation (manual)! Getting started was really simple, and I quickly understood what it can and cannot do.

"Bombadil" means that I'll probably skip most of these tests.

Very cool stuff, will apply this to my next project

From your title my immediate thought was "cool, maybe this will move us a bit closer to making components (or the testing thereof) cover accessibility thoroughly by default".

See, the idea with the semantic web, and the ARIA markup equivalents, is that things should have names, roles, values, and states. Devs frequently mess up role/keyboard interaction agreement (example: role=menu means a list will drop on Enter keypress, and arrow keys will change the active element), and with ensuring that state info (aria-expanded, aria-invalid, etc.) is updated when it should be.

Then I checked the Antithesis website. They don't even have focus state styling on any of the interactive elements. sigh

For most static UI surfaces, I probably wouldn't use it, but I can see a use case in this for testing generative UI workloads.

Struggling to understand what this is or how it works.

very cool! does this work? can you describe the kinds of real bugs you've caught with this?

Bombadillo Crocodillo

Ok I will see myself out

(Yes I know it's actually from the Tolkien book)

Timely! I spent a good chunk of yesterday investigating Bombadil. I'm a huge fan of PBT and a huge fan of temporal logic (though not linear temporal logic) and we're currently trying to level up our QA, so it seemed like a good match.

Unfortunately, I concluded that Bombadil is a toy. Not as in "Does some very nice things, but missing the features for enterprise adoption." I mean that in a very strong sense, as in: I could not figure out how to do anything real with it.

The place where this is most obvious is in its action generator type. You give it a function that generates actions, where each action is drawn from a small menu of choices like "scroll up this many pixel" or "click on this pixel." You cannot add new types of actions. If you want to click on a button, you need it to generate a sequence of actions to first scroll down enough, and then look up a pixel that's on that button.

Except that it selects actions randomly from your list, so you somehow need the action generator, when run the first time, to generate only the scroll action, and then have it, when run the second time, generate only the click action. If you are silly enough as to have an action generator that, you know, actually gives a list of reasonable actions, you'll get a tester that spends all its time going in circles clicking on the nav bar.

(Something in the docs claimed that actions are weighted, but I found nothing about an actual affordance for doing that. Having weights would make this go from basically impossible to somewhat impossible.)

(Edit: Found the weighted thing.)

I am terrified to imagine how to get Bombadil to fill out a form. At least that seems possible -- you can inspect the state of the web page to figure out what's already been filled out. But if you want the state to be based on anything not on the current page, like the action that you took on the previous page, or gasp the state on disk (as for an Electron app), that seems completely impossible. Action generators are based on the current state, and the state must be a pure function of the web page.

Its temporal logic has a cool time-bounded construct, but it's missing the U (until) operator. One of their few examples is "If you show the user a notification, it disappears within 5 seconds." But I want to say "When you click the Generate button, it says 'Generating...' up until it's finished generating." And I can't.

(Note: everything above is according to the docs. Hopefully everything I said is a limitation of the docs, not an actual limitation of the framework.)

I shared my comments with the author yesterday on LinkedIn, but he hasn't responded yet. Maybe I'll hear from him here.

I have a pretty positive opinion of Antithesis as a company and they seem to be investing seriously in it, and generally see it as a strong positive sign when someone knows what temporal logic is, so I have hopes for this framework. I am nonetheless disappointed that I can't use it now, especially because I was supposed to finish an internal GUI testing tool this week and my god I'm behind.

My kingdom for a way to stop this godforsaken industry from stripping Tolkien's fiction for parts.