Astra for Law

I know someone who works in law and deals particularly with an area of US benefits and healthcare law. One of their workflows for lower-level employees at their firm involves taking in documents from healthcare plans and organizations, analyzing them for certain kinds of data, and then importing that data into an internal system they use to analyze and provide guidance on plans. The internal system can contain hundreds of documents for an individual client. All of the documents have the same information (roughly) but in totally diverse formats and styles. Once it's in the system, it's easy to compare and analyze across documents and the research process is much faster.

They recently bought a Claude subscription and began using Claude to do the initial read of the documents and output JSON they can import into their internal systems. The work still must be reviewed by an attorney - Claude is nowhere near making the kinds of judgments a lawyer would make about this content - but it has increased their throughput from 2-3 documents an hour to 8-10 documents an hour by killing the busy work.

LLMs have great advantages for this kind of work - but not for decision-making. I just don't see OpenAI ever admitting that.

(I've left some details intentionally vague because this is a very specific area of law and I don't want my friends to be identified without their consent.)

Lawyer here (non practicing so to be clear none of this affects me):

most comments I read here don't seem to realize that different areas of law have very very different economic models and don't even mention which one they think will be affected or why, they just sort of lump it all together.

For example: It is highly unlikely llms will have any meaningful effect on high value personal injury law - I don't see a 5 million dollar case being handed to an LLM when the majority of the cost is in trial aids and not even lawyers. It may affect where and how they advertise. It may affect how they work. But it seems really unlikely to put any of them out of business any time soon by people doing it themselves.

Will it affect other areas more? Maybe. Probably? But so far I haven't seen a ton of comments that make specific enough arguments that they could really be debated or responded to effectively with a useful opinion

The need for actual lawyers will persist I think from my own experience. I attempted drafting a contract with some points myself using AI, but after several edits I wasn't sure if it was correct. Sending it to an actual lawyer ended up in so many corrections I couldn't imagine the first time. One big thing was the overly excessive protective clauses which didn't make sense for reality or conflicted with another.

Its just like code I suppose, if you can read and understand and validate, you can use it to scale and otherwise it could end up being a vibe effort.

Second paragraph:

> API customers including Harvey and Legora will be able to build on Astra for Law, bringing this intelligence into their own products and workflows.

In other words: "no, no, we're not eating our children to prep for the IPO. Don't worry."

The courts are about to overrun with AI-generated lawsuits (even more so than they have been[0]).

[0]https://www.technologyreview.com/2026/06/04/1138391/courts-c...

> At the highest reasoning effort for both systems, Astra for Law passed the evaluation’s overall correctness check on 54.0% of questions

isn't that kind of poor?

No word on model hallucinations in the blog post.

https://artificialanalysis.ai/models/gpt-6-astra?omniscience...

"We have dangerous AGI that can destroy humanity."

"Also, all of your sensitive legal documents will be totally safe with us."

"Also, for some reason even though we have AGI and selling tokens is a fine business, we need to sell a new product specifically targeted at a very high margin and lucrative industry."

I'm not familiar with the Vals AI Legal Research Benchmark. But their website has other frontier models' scores, and the scores OpenAI is now revealing for "Astra for Law" are slightly less than Claude and Muse:

> The top is a three-way tie: Muse Spark 1.3 Max, Claude Opus 5, and Claude Fable 5.1 all reach 55.29% all-pass accuracy, a clear ~6-point step ahead of the next model. [Astra for Law reached 54.0%]

> Under partial-credit scoring, Claude Opus 5 reaches 90.58% weighted pass rate but 55.29% under strict all-pass grading, where every rubric check must pass. The gap shows models often get most of an answer right but fail on one or two required elements. [Astra for law reached 90.0%]

https://www.vals.ai/benchmarks/legal_research

Really, what could go wrong?

How long before we start building detailed models of each judge trained on all of their legal output, and then test various legal theories against those judge models in virtual moot court? Craft each pitch to the legal idiosyncrasies of the batter. I assume that real lawyers do this routinely and could use a simulator.

open-ai have proven they can make good / decent models but business strategy is just spray and pray.

they need to pick a lane and optimize for it. coz at their size they can't serve the application layer (a.i startups who can fine-tune models will eat their lunch)

if they gonna do a consumer play - then go ham on that.

otherwise they're gonna get caught in the dreaded middle valley.

So OpenAI is partnering with Latham Watkins, Freshfields is partnering with Anthropic and Kleiner Perkins is building their own. It'll be interesting to see which wins out here, I don't see how those partnerships can end well for the law firms unless they're making an assumption they'll be sucked dry of USP but the revenue split from the AI labs will make up for it. Why would I pay a premium for Latham Watkins when every other firm can get their expertise and experience in a subscription, and add their own on top?

> Astra for Law passed the evaluation’s overall correctness check on 54.0% of questions.

How is this a product that you are selling?

memes and snark aside, can you use this to create legitimate terms and contracts for my products and if I do who is getting sued when it is wrong?

The most interesting part about this to me was how they bench/compare it, like in the example with Fable:

  "Given the same prompt, Astra for Law returned two closely matching precedents; in the litigation example, Claude Fable 5.1 returned a holding that had been reversed on appeal, while in the transactional example it reported finding no such case."

It made me wonder if a good deal of law is about finding a way to work in statements with clear precedents without your opposition noticing and then later drawing upon them in court (as settled precedents, in your favor) after the opposition (perhaps implicitly) accepted it. That would clarify a lot about why some lawyers need to spend so much time pouring over and memorizing past cases (even ones that are only tangentially related); because anything they miss could be used as a potential trojan horse by the opponent.

If this is true that must mean there are a good deal of cases settled using precedent "gotchas" where both sides knew that without the "load-bearing" precedent the outcome would've definitely been the opposite. (i.e precedents almost always trump even valid arguments)

Can you imagine the sort of corruption that is possible here? Like if you have a lawsuit against someone or some entity that openai or their investors have business with...

In an utopian society, lawyers are an unnecessary profession. Laws should be clear and simple so the common person can be their own "lawyer". LLMs help with that goal.

Any lawyers here who have used AI agents heavily for their work? From what I've heard, they're currently very good at searching, analyzing and drafting documents like contracts and patents, but some say they suck at interpreting the law.

Recently was involved in legal matters requiring lawyers - I used OpenAI extensively for research, advice etc - I have to be honest, much of it was completely useless - the real world outcome with lawyers in the room was very different from what OpenAI was spitting out.

I wonder if Lawslop is gonna become a mainstream expression. Anyone got a better term?

So much for caring about the spirit of the law. Now we'll start an arms race for abusing every possible letter of the law.

It's analogous to crypto. Started from some noble anti-authoritarian ideas and morphed into machine that removes any friction for capital - whoever has the most money will keep gaining the most.

I imagine the point here is to separate out the APIs by different professions and charge accordingly.

Will the legal fees reduce after this? How about training AI models with laws of other countries especially the democratic ones? This is where sovereign AI models are required otherwise they will start hallucinating wrong laws of different countries.

I remember OpenAI was talking about sovereign AI models (OpenAI for Countries), it is time to train Astra India with Indian Constitution.

Seems like another “product” that will be killed in 6 months, but is good for the IPO so they can say they solved law.

what is going on here?

this vertigoruntime is brand new and his submissions absolutely dominate the front page recently https://news.ycombinator.com/submitted?id=vertigoruntime

the about link https://vertigo.kuber.studio is obvious AI llm spam

An important take-away from this is the harness is really important. The graphs show that the same model with a better harness performs many percentage points better. OpenAI are getting into the "selling the harness" game in a big way. Why?

What's interesting about that is while not everybody can train or run a model, anybody can build a harness. You and I can build harnesses.

It seems strange that OpenAI would move into a field where any developer can compete with them. I think that tells us a lot about the economics of training and selling inference.

"Rogue OpenAI agent swarm accidentally overturns the Civil Rights Act"

followed a month later by

"Anthropic's Claude inadvertently repeals the 19th amendment"

What we contributed to Reddit and Arxiv voluntarily before, what we contribute as traces right now are all being used to build business verticals by OpenAI and Anthropic. All these business verticals are being used for is more trace collection which would only strengthen these models and render most humans useless because the ceiling for 1000 swarms of Agents to learn is much higher than an average human. These companies clearly can identify the relevant traces for these business verticals which only strengthens the argument that NS was built on someone else's traces.

Just like breaking crypto in the age of cloud is more about cost than time, this will lead to legal attacks based on the same principle. The biggest wallet wins.

This seems like a very similar set of tools to Anthropic. But I’ve enjoyed using the various frontier models to criticize each other.

Using recursive loops, the output has gone from a high school level intern to a 2nd year lawyer in about a year. It still doesn’t beat the experts, but so much legal work is (legally significant) pedantry, not legal philosophy.

AI will not kill off lawyers, or reduce the amount of litigation. It will increase volume and velocity.

Can someone explain to me the economics of AI and how it intersects with billing by the hour?

I think there would be a strong incentive not to use tools that speed up your work because you'd effectively be able to bill less time?

I'm sure there are some firms out there with more work than people, but still wouldn't it be more effective to hire another human who can then bill at a high rate for many hours?

It’s a bad day to be a lawslop company. When you’re reliant on other companies to do all of the AI part of your AI product, they can just train on your traffic and eventually eat you.

I have always felt like LLMs are uniquely suitable for legal work. I really have trouble believing we will have hardly any legal assistants and paralegals going forward when these LLMs are so incredible at spotting issues with arguments, figuring out citations, and doing semantic search.

Legal domain is a challenge for most firms, even ours. This is a true game changer even if it looks a bit slop style in the output, the immediate uplift is quite massive.

The domain remains hard due to the lack of availability of high quality LLM-ready data providers in legal space.

I find the watermarking dynamic to be really interesting in the legal space, as more large model providers provide increasingly powerful legal capabilities, and adoption (presumably) also increases. Attorneys aren’t the same as developers as their work can be traced back to them, and there are personal bar licenses and reputations at stake. I wonder if knowing the likelihood that AI generated something helps or hurts in that respect.

I also see a lot of watermark removal services popping up as a result.

The most interesting use case in my mind is skipping law suits. Obviously you need lawyers in court. But lawyers are people you are basically paying to fight for you.

Instead, if you resolve your dispute outside of court, you don’t need a lawyer. If both parties use ChatGPT to find the relevant laws or read contracts, they could come to an agreement without expensive legal fees.

There is a lot of stuff here that I don't understand, but the concept of law firms giving user reviews is quite funny to me. Those reviews are going to be the most non-legally binding reviews ever written lol.

"Felt like a significant step toward legal-focused AI."

"Showed strength across key aspects of legal research."

Will it be able to sue itself?

Since the cost of building software is now cheap, there is nothing stopping them from building everything imaginable. They'll soon have an app store with every app built by them and they'll say its for security reasons. Nothing is stopping this coming monopoly

The biggest problem with this is billable hours. Faster work means less billable hours for attorneys.

I do believe entry-level paralegals will be made obsolete. In the grand scheme, perhaps it reduces overall cost of legal assistance, which is a net benefit for society.

From my experience LLMs seem to forget sometimes who they are representing when drafting clauses or editing / redlining.

Our counsel made a few edits where it clearly drafted in favor of the customer instead of us.

Surprised they published this without mention of jurisdiction. Each country has their own laws.

Would have been prudent to highlight that this is for (presumably) US law

It's going to be hard for mediocre lawyers to survive now... I wonder what it takes to survive these days?

We need a term for the dark pattern of zooming into just that part of the y-axis where the two closely competing benchmarks sit, to make the top one appear maximally better.

Interesting to see the callout to companies like harvey in the post itself as consumers rather than competitors? I guess openai isn't quite willing to step into those customer relations themselves?

So there's a strategy shift here. They launched financial services specific tools and now law?

Is the play here a set of specialized harnesses using their best general model?

surprised the word RAG hasn’t come ho in this thread (except for a likely-LLM-generated-and-therefore-downvoted comment).

>with settings, tools, and context

call me crazy, but I think that this kind of suite, which training-uber-alles people generally dismiss as trivially replicable ‘wrapper’ is actually the differentiating factor for LLM adoption today and moreso into the future.

I’m not dismissing the near-all-out impact of training, but from a competitive busines or industry-structure lens, we’re looking at the three or four big players competing their utmost ultimately, if unintentionally, to turn foundation model access into commodity.

To the capabilities-maximalist minded (typical among engineers - my former life so I’m familiar don’t lack guilt in committing that) folks who will say “Oh the foundation model megacorps will just build out any wrapper whenever one of their third party wrapper plays demonstrates enough adoption, my rejoinder:

Apple did not rebuild an Uber-like app and cut Uber out.

We’ll see how this OpenAI legal services industry wrapper plays out, but I suspect 1) the third party legal wrapper plays will run to other foundation models not doing a legal wrapper, and 2) 3rd party wrappers will do a better job of it since it’s their all-out focus, unlike OpenAI’s whose priorities are necessarily more generalist.

Yes, we all remember the breakout startup failure-arguing quote “Google has entered your space.” That worked for several high profile applications. I believe more of those bets died on the vine than broke-out succeeded however, we only remember the biggest ones that persisted.

If legal services AI turns into one of the Mail or Maps-scale applications of the AI industry, while that would be a fair strategic action counter to the thesis I’ve laid out, the thesis itself would still tolerate it. It’s a question of short-fat tail vs mid-to-long-tail application scope & attractiveness. For example, I think it’s clear that coding is one of these short-fat-tail applications, and the low-no code plays are absolutely having their lunch eaten to acqui-hire ‘death’. I just doubt that the same will persistently transpire facing all professional service wrapper plays.

Post AI era view on anything can be categorized as “What can go wrong ?”

For law, then medicine, mathematics, physics, etc. I believe LASSI Local Artificial Super Specialized Intelligence is the future, just before it becomes GODD General Omniscient Distributed Daemon

The end goal should be AI judges, I think China has implemented that to some degree.

This is the first blog post I saw OpenAI call out Claude directly like that.

I've had top SV lawfirms whos partner charged our company $2000/hr and still couldn't get the right docs in the signature packet. and another getting share counts wrong during raise.

frustrating that law firms have no liability for these mistakes

I welcome ai law

Astra isn't doing well on bullshit Benchmark https://petergpt.github.io/bullshit-benchmark/viewer/index.n... and is even worse in Legal department.

Qwen 3.8 Max and Opus 4.8 score highest.

Revealing how they call Harvey and Legora customers instead of partners...

Everyone involved is about to get a swift and thorough introduction to the world corporate contract lawyers. But don't worry, they're extremely sympathetic and understanding when you flub a multi-million dollar corporate contract and will happily refund that money.

As a user of ai hooked up to my Gmail I assume I’ve lost all privilege

Went from AI replacing my job as an attorney, to them begging me to use their tools

yes, most of the time i use Astra Low already.

Espn has an article about Lane Kiffin using this for his legal advice recently.

He almost singlehandedly killed LSUs football program until people were able to convince him to talk to a real lawyer.

yes, law decision depends on human

This is really cool.

Make em Grant Bail. Make no mistake.

I really hope the bar associations continues to hold lawyers to high standards but I have feeling they may not be ready to handle fallout of AI slop-law.

People that are saying OpenAI is screwed because a lack of profit, I'm not of that opinion. They are encroaching on every industry they can. They have name brand recognition, a huge user base and are showing they can be a valuable tool to all types of businesses.

As much as I hate to see it. They are now threatening industries like Engineers, Game Developers, Accountants, 3D modelers, 3D animators, Video Production, Audio Production, Therapist, Tax Auditors, Journalists, Authors, Artists, Mathematicians, Product managers, Every type of analyst and pretty much any other job that can be done behind a computer screen.

We have big problems for humanity.

Wouldn’t this and similar efforts to centralize bureaucracy make AI the new gatekeeper? Without reliable transparent models we’re just trusting OpenAI instead of a hundred top legal firms.

I said this before but I wonder if Dan Kan will reboot Atrium. Rally up some old partners and hope Anthropic buys them out for a couple billion.

where are the benchmarks

how about "just turn off your company if you're so worried about AI instead of immediately suggesting we use it to judge people's lives and livelihoods"?

this essay details how law firms became sweatshops from 80s. They charge hundreds of dollars to do make busy work by the junior most staff. LLM will kill the goldengoose of the law industry.

https://aeon.co/essays/what-made-law-into-a-white-collar-swe...

Will these models eventually replace all knowledge work, leaving lawyers, doctors, product managers, software developers, and others out of a job?

If the benefits were shared across humanity, that could bring us closer to utopia. My worry is that we’ll instead end up with a handful of even wealthier billionaires and millions of people out of work.

"Introducing Astra for Dog Walkers"

Cool, now you can pay an attorney $500/hr for them to prompt Astra for 30 minutes, and bill you like they spent the normal 8 hours on the case.

The grifts continue... imagine something as consequential as Law being advertised as being solved by a statistical word generation engine that regularly gets basic things wrong. Anyone who isn't a lawyer won't know any better but you draft a single document of any appreciable detail and send it to an actual lawyer and it's littered with problems.

[flagged]

[dead]

[dead]

[dead]

[dead]

Law is a solved problem.

I've always said this will be when we get the real Butlerian Jihad, when the AI firms start trying to liquidate the legal profession.

If you automate lawyers out of a job, you can absolutely automate lawmakers out of jobs next. (Not that this would be a bad thing? Maybe pervasive agents for everyone can be the gateway drug to a "this time it's different!" workable direct democracy)