- cross-posted to:
- technology@lemmy.world
- news@beehaw.org
- cross-posted to:
- technology@lemmy.world
- news@beehaw.org
You must log in or register to comment.
Easy - detect if you’re getting accessed by a search crawler or a human. Serve a full page or just a login request.
So how can a user pretend to be a web crawler?
This browser addon can spoof useragents:https://add0n.com/useragent-switcher.html
You’re going to need a special hat.
Perhaps a safety vest as well. You know, for access.
Adding a clipboard and a ladder will make it even more official
Ever heard of https://12ft.io/ ? It allows you to bypass alot of pay walls by basically pretending to be a search engine trying to index a website. For SEO reasons a lot of pay walled sites allow search engines to access the whole article to index. 12ft.io leverages this to show you whole articles behind paywalls. This is something you could also achieve by spoofing the User-Agent. It would probably work for things like Pinterest without an account as well, but that’s something I have never tried (since I have no interest in the cancer that is Pinterest).
