If you have real traffic and bot traffic, you still need to identify which is which. On top of that, bots very likely don’t reuse the same IPs over and over again. I assume if we knew all the IPs used only by bots ahead of time, then yeah it would be simple to blacklist them. But although it’s simple in theory, the practice of identifying what to blacklist in the first place is the part that isn’t as simple
Why not simply blacklist or rate limit those bot IP’s ?
Many bots cycle through short DHCP leases on LTE wifi devices. One would have to accept blocking all cell phones which I have done for my personal hobby crap but most businesses will not do this. Another big swath of bots come from Amazon EC2 and GoogleCloud which I will also happily block on my hobby crap but most businesses will not.
Some bots are easier to block as they do not use real web clients and are missing some TCP/IP headers making them ultra easy to block. Some also do not spoof user-agent and are easy to block. Some will attempt to access URL's not visible to real humans thus blocking themselves. Many bots can not do HTTP/2.0 so they are also trivial to block. Pretty much anything not using headless Chrome is easy to block.
True, most of the blacklists systems today aren’t realtime like Amazon WAF or Cloudflare.
We need a Crawler blacklist that can in realtime stream list deltas to centralized list and local dbs can pull changes.
Verified domains can push suspected bot ips, where this engine would run heuristics to see if there is a patters across data sources and issue a temporary block with exponential TTL.
There are many problems to solve here, but as any OSS it will evolve over time if there is enough interest in it.
Costs of running this system will be huge though and corp sponsors may not work but individual sponsors may be incentivized as it’s helps them reduce bandwidth, compute costs related to bot traffic.
In the real-time spam market the lists worked well with honest groups for a bit, but started falling apart when once good lists get taken over by actors that realize they can use their position to make more money. It's a really difficult trap to avoid.
How? They run their scraping and training infrastructure - and models themselves - from within those “AI datacenters”[1] we hear about in the news - and not proxying through end-users’ own pipes.
[1]: in quotes, because I dislike the term, because it’s immaterial whether or not an ugly block of concrete out in the sticks is housing LLM hardware - or good ol’ fashioned colo racks.
After the first five minutes of using it on Ubuntu, it crashed with error saying I don't have enough free memory, quick look into system stats proved that wasn't the case.
Anyway, not a great first impression. I guess I'll try again in a few months.
I'm not a big Excel user, but I see errors that I get when I type in English function names while using a non-English version of Excel. Is it correct that functions (and thus this xlsb file) are not portable to other language versions of Excel?
It's been my experience that Excel spreadsheets are not transferable from one locale to another. Maybe there is a "culture-invariant" version of a spreadsheet but I haven't found it.
Even the CSV export of Excel sets the separator based on locale, rendering them hard to use in international setting. I work at a European statistics office, and although there's SDMX, it's not under Save-as in Excel.
In US, you can find it on Amazon for 15-20 bucks.