deleted by creator
Until they ramp up their own AI money making projects.
I’d rather like it if they train it on stuff I say. I want the AI of tomorrow to reflect my thoughts.
seriously I would much prefer gold tier journalism and news sites let it crawl so when people use it to make choices in the future they’re guided to better choices.
it is honestly so hard to know what will happen though, it’s so complicated it’s virtually guaranteed we’re not correctly anticipating the consequences of any of this. I’m not really even talking about the AI, I’m talking about the effects on society which are a lot more complex.
It’s just about the money really, they want their cut of the AI money cake.
But for large website operators, the choice to block large language model (LLM) crawlers isn’t as easy as it may seem. Making some LLMs blind to certain website data will leave gaps of knowledge that could serve some sites very well (such as sites that don’t want to lose visitors if ChatGPT supplies their information for them), but it may also hurt others. For example, blocking content from future AI models could decrease a site’s or a brand’s cultural footprint if AI chatbots become a primary user interface in the future. As a thought experiment, imagine an online business declaring that it didn’t want its website indexed by Google in the year 2002—a self-defeating move when that was the most popular on-ramp for finding information online.
Really curious how this will end up
That’s an interesting point that I hadn’t considered, the comparison to Google indexing in the early 2000’s may prove to be very apt with the number of people I’ve seen using chat GPT as a search engine.
I’d bet sites blocking ChatGPT will regret it when (not if) Bing starts using it for search engine relevance.
That’s because you block the GPT crawler doesn’t mean you are no longer indexed
Serious question — you think any amount of AI will make people use Bing? 🤔
I started using it this year because its actually been giving me decent results unlike google. We’re…in a dark timeline
Hmm. I have an incredibly strong aversion to everything Microsoft, so even giving Bing a chance is difficult. However, I must admit that I can recognize the part about Google not delivering. I even went so far as to tamper with the CSS recently just to make Google’s results slightly easier to parse.
Maybe it’s time to try something new 🤔 I just wish the only viable alternative wasn’t made by Microsoft 😓🤢
Dark timeline, indeed! 😔
Check out Kagi, paid search is extremely worth it. Stop being a product to sell and start being a customer.
Is it possible that they offloaded the scraping to a different company to avoid direct litigation now theyre out in the open? To say “we didn’t scrape your website, and you can’t prove it.”
Like DDG, Ecosia, Qwant use Bing for their data Or how feds buy data from data brokers. Outsource the dirty job like every tech company does and shift the blame if caught doing something unlawful.
It seems they are trying to garner some positive PR after they scraped through everything without anyone noticing.
I absolutely believe a lot of companies outsource simply because they don’t want to build an internal organ to do it. Even in government, despite what Conservatives believe, most organization heads are pretty focused on core competency and press to use outsourced resources. This latter also promoted by heavy lobbying by the companies selling the services.
This is a situation of “never attribute to malice that which can be easily explained by stupidity.” Sure, some are motivated by malice or subterfuge, but most are probably just buying services because they have other things they’d rather focus on.
Like it is useful… Open ai already got all the useful info out of the websites.
Tho maybe for the sites generating new content it may have a use. But all the content before that is already lost to chatgpt.
“Lost to ChatGPT” is a weird way of putting it. The content is still there, nothing’s happened to it.
You know what he means. My data is “lost” to google, I’m sure you’d agree with that.
Lemmy.ca added a block at the nginx level for it
https://lemmy.ca/comment/1999439
# curl -H 'User-agent: GPTBot' https://lemmy.ca/ -i HTTP/2 403Hilariously, unless ALL lemmy instances do this, anyone that federates with you will have to block it too or any communities they sync with you will be available on their instances…
I know but at this point you do what you can.
Yeah, it’s already too late. Why didn’t they provide this before they already scraped websites?
You think Google thought about
robots.txtbefore they developed their search engine? Nah, it’s all public Internet, and they scraped away.A non-zero percentage of web sites will bother to follow these instructions, but it might as well be zero.
Very early on, at least, their spiders respected robots.txt.
I know there are folks that have all of the Big G in their robots.txt files on principle, might want to ask them if it works or not.
I do and I can confirm there are no requests (except for robots.txt and the odd /favicon.ico). Google sorta respects robots.txt. They do have a weird gotcha though: they still put the URLs in search, they just appear with an useless description. Their suggestion to avoid that can be summarized as: don’t block us, let us crawl and just tell us not to use the result, just trust us! when they could very easily change that behavior to make more sense. Not a single damn person with Google blocked in robots.txt wants to be indexed, and their logic on password protecting kind of makes sense but my concern isn’t security, it’s that I don’t like them (or Bing or Yandex).
Another gotcha I’ve seen linked is that their ad targeting bot for Google AdSense (different crawler) doesn’t respect a
*exclusion, but that kind of makes sense since it will only ever visit your site if you place AdSense ads on it.And I suppose they’ll train Bard on all data they scraped because of course. Probably no way to opt out of that without opting out of Google Search as well.
Now that’s a dirty trick.
Yeah I always assumed robots.txt only told them to hide it from search results, but Google still scrapes everything they can from you. The illusion they skipped over you
If you look in the server logs, you can see what their spiders are grabbing.
