I run a service[1] that was getting hit pretty hard by these crawlers.
Ultimately instead of going down this path, I decided to just start charging for access to the service (it was long overdue)[2].
Users who are logged out can still see old cached content (which is a single DB read op), but to aggregate new content requires an account. I feel like this is a good (enough) middleground solution for now.
Ultimately instead of going down this path, I decided to just start charging for access to the service (it was long overdue)[2].
Users who are logged out can still see old cached content (which is a single DB read op), but to aggregate new content requires an account. I feel like this is a good (enough) middleground solution for now.
[1]: https://kulli.sh
[2]: https://lgug2z.com/articles/in-the-age-of-ai-crawlers-i-have...