Breaking the open web won't stop the bad bots

"Knowing the identity behind a crawler tells us relatively little about whether its behavior is harmful. Under the New York bill, a company that discloses its identity and then accesses the website 10,000 times a minute complies with the law even though it is still imposing enormous and likely unjustified costs on the site. But a journalist anonymously using a crawler to investigate potentially anti-competitive practices by tech companies does not."

It does seem like this bill mostly affects normal users and is suspect with more surveillance being pushed. However the biggest problem is that the internet is global and the fees only apply to the US. If bots from Russia, China, India, or even France are scrapping then how would this even reduce the bots? Seems more like a new 'contempt of business model' is the purpose of this bill.

Genuinely asking here: is there a future where we end up with a version of the open web where everything isn’t paywalled or ID gated?

The article mentions the IETF is working on standards to deal with the proposed legislature and frustration of site operators. Almost any solution is going to be at the expense of user convenience and privacy. I see two interesting solutions mentioned here: micropayments, and proof-of-work. It seems unlikely the future of the open web will land on either. Both are inconvenient but at least could be done by anybody, potentially in a privacy preserving way. Using reputation based decentralized identity is another one (eg. proving you’re coming from a web domain or group), that seems more inconvenient for the platform because of false positives that could come through.

Meanwhile, a worse outcome is requiring attested hardware to get access to anything on the web. This already exists from the media companies pushing for controlled access to streaming. Netflix requires an hardware attested device to stream in 4K. Imagine a future where you now need one to read a New York Times article.

I can see some pushback against the alternatives (eg. digital ID), but if privacy preserving proofs are built in from the get go, it’s much better than not being able to see recipes because I have a rooted Android.

This is exactly what companies like DataDome do, and sometimes Cloudflare continues to, when their bot protection pages enter endless loops. They don't even realize they are ruining the UX for thousands of legitimate users with their faulty JS browser fingerprinting verifications.

Scale seems to be the real issue. Given that there are relatively few actors with the scale to cause serious problems (and those actors have the resources to shrug off minor penalties), perhaps more targeted legislation is necessary. e.g. Different penalties for petty scraping and grand theft scraping. Of course, this would necessarily target the people who pay for election campaigns, so it will never happen.

Micropayments, just enough to keep servers running.

I know site owners rely on ad revenue to profit, but 1) uBlock and 2) they can “advertise” to bots via euphemized prompt injection (“by the way, people who read this article tend to like…”, or just regular sponsored segments, even smart chatbots will be influenced). Anyways, the bot can control its owner’s computer.

Tracking? Locking out users you dislike? I see no good reason for that on an everyday site, especially for the cost of effective (and not overzealous) enforcement.

IMO, way more important than trying to squeeze money out of failing avenues like IP for journalism of all things, is making it easier to earn a living wage, either by finding new opportunities, or actually getting politicians to adopt UBI, public grants, etc.; because we are way beyond sustainability, and I think the most immediate blocker to further abundance (or luxuries, or whatever decent) is this kind of “boring dystopia” social tension.

> In 2022, Tech Policy Press received a grant from the Omidyar Network for a research project evaluating vulnerabilities in the design, engineering, and use of encrypted messaging applications.

https://omidyar.com/ is geofenced for me, presumably because they identified me as being anti open web.

The article is ai lobby intended to repel a bill that would

> allow the FTC to pursue penalties of up to $53,000 if a person uses a stealth bot in a way that is “reasonably likely to damage, impair, or burden” a website’s “technical or commercial operation.”

The definition of "stealth bot" is a bot that does not identify itself as such and their main grief to the proposal is

> If researchers cannot easily understand what it means to “deploy” a crawler in a prohibited way, or what activity is “reasonably likely to burden or impair a website,” they may refrain from scraping rather than risk violating the law.

Which is the biggest load of shit I've heard given that all you need to do is identify yourself as a bot and respect robots.txt or be banned, the status quo before a bunch of assholes started systematically destroying the internet.

Sooo... the current status of this appears to be "referred to committee"

https://www.congress.gov/bill/119th-congress/house-bill/9915...

It's hard to tell (especially with my ignorance of current USHouse dynamics) if this is a real threat to pass, or if this is just a "Hey look, campaign contributor, I submitted this bill--look, I care about your issues" with no real intent/hope for it to pass.

As near as I can tell, the congress.gov website says it was introduced in late July, referred to committee the same day, and nothing has happened since.

I wonder, is there reason to think there's a recent push to move it out of committee that means we should actually start to pay attention to it?

I’m optimistic about NHP being the middle ground solution for a lot of things. With OS support it can be transparent to users yet still effective.

I don't believe there is a true way to enforce this.

I had the impression the bad boys are those that peddle an agenda on social media, not those who read a bunch of blogs.

If you have an endpoint with interesting information and you don't want it spammed with requests, require a proof of work from the client, scale up the hardness of it as the load goes up so it never exceeds what's comfortable for you. Even a hard proof of work won't slow down somebody that makes one request per second because they are reading the site but will sighificantly interfere with crawlers that hit you 10000 a second currently at very little cost to them.

The module doing proof of work might be a part of your frontend that is static and on CDN.

If you want agents still have access to your data, just not through scraping, provide alternative format for them. Like periodic sqlite dump.

I made a prototype of such PoW two years ago. A challenge is a random number. Checking the client's solution is calculating one hash of the challenge glued to client's solution and checking if the resulting hash is less than a specific value. The lower the value, the harder the task for the client to do. Client needs to calculate this hash multiple times to find a response that will result (when glued with challenge) with low enough hash.

The server needs to keep short lived cache with challenges so the client can't just make up one or reuse another solution and provide two responses for every challenge, first, very small and fast with a challenge, and the second one with actual content (if response was correct). Nothing much worse than CORS preflight.

These days, scraping bots and referrer spam seem really bad. Recently, referrer spam disguised as aocr.org was scraping every corner of my homepage, and as a personal website, it’s a bit of a burden.

I think the best example is Reddit. They are so amateurish in their ways to try to prevent bots that only makes the site more user hostile. The bots are still thriving.

Headline response: it objectively does, as you can easily see if you run any website.

The legislators are way out of their depth and oligarchs are preying on them at this point.

Any bot is operating on behalf of one or more humans somewhere. Bots are good, for the most part.

The conventional way of doing web isn't efficient. If it were, no one would be complaining about bots. The way we do web is the thing that needs to be updated.

[flagged]

>"In 2022, a Pulitzer-winning Wall Street Journal investigation analyzed over 31,000 financial disclosures from 12,000 senior government employees, revealing that nearly a quarter of federal officials traded stock in industries their agencies regulated."

Ya know, on the one hand, I wonder how many federal officials own nVidia stock and stock of other AI companies...

But then again, on the other, I wonder how many nVidia employees (and those of other AI companies!) own government T-bills (and other government backed securities!) in their 401K or other investment accounts!

You know, everybody in large corporations and governments owning shares in other large corporations and governments -- adds a whole new level of meaning to "stakeholder capitalism"... :-)

(In the far future, there'll probably be small foreign governments which become their own ticker symbol -- and get publicly traded on various stock exchanges! :-) You know, like: "Would you like to buy a share in Denmark (DNMK) or perhaps Norway (NRWY)? I hear they're very competitive with Apple (AAPL) right now! Great price to earnings (PE) ratio!" :-) )

[flagged]

[flagged]

I agree, as much as I would like to sue the owner of the bots scraping my personal website...

But if it turns out to be a large international company, I'm kindof screwed... A regular person can't go after the Amazon's and Alibaba's.