But for large website operators, the choice to block large language model (LLM) crawlers isn’t as easy as it may seem. Making some LLMs blind to certain website data will leave gaps of knowledge that could serve some sites very well (such as sites that don’t want to lose visitors if ChatGPT supplies their information for them), but it may also hurt others. For example, blocking content from future AI models could decrease a site’s or a brand’s cultural footprint if AI chatbots become a primary user interface in the future. As a thought experiment, imagine an online business declaring that it didn’t want its website indexed by Google in the year 2002—a self-defeating move when that was the most popular on-ramp for finding information online.
Really curious how this will end up
That’s an interesting point that I hadn’t considered, the comparison to Google indexing in the early 2000’s may prove to be very apt with the number of people I’ve seen using chat GPT as a search engine.
I’d bet sites blocking ChatGPT will regret it when (not if) Bing starts using it for search engine relevance.
That’s because you block the GPT crawler doesn’t mean you are no longer indexed
Serious question — you think any amount of AI will make people use Bing? 🤔
I started using it this year because its actually been giving me decent results unlike google. We’re…in a dark timeline
Hmm. I have an incredibly strong aversion to everything Microsoft, so even giving Bing a chance is difficult. However, I must admit that I can recognize the part about Google not delivering. I even went so far as to tamper with the CSS recently just to make Google’s results slightly easier to parse.
Maybe it’s time to try something new 🤔 I just wish the only viable alternative wasn’t made by Microsoft 😓🤢
Dark timeline, indeed! 😔
Check out Kagi, paid search is extremely worth it. Stop being a product to sell and start being a customer.
Lemmy.ca added a block at the nginx level for it
https://lemmy.ca/comment/1999439
# curl -H 'User-agent: GPTBot' https://lemmy.ca/ -i HTTP/2 403Hilariously, unless ALL lemmy instances do this, anyone that federates with you will have to block it too or any communities they sync with you will be available on their instances…
