@feld @grillchen @kaia This is more a matter of consent. If someone puts up a thing on a blog on the internet you can usually safely assume they wanted to be seen by others. Otherwise they would have taken measures to prevent you from seeing it.
But yeah from a legal standpoint yeah it can be infringement. But rather on the basis that you actually make a copy of the website on your machine (caching and stuff. There is a long list of legal battles on that subject with video streaming lol)
Even assuming everyone is fine your crawler seeing things they might not be fine with any other uses.
The question really becomes is scraping the web to create a learnset for an LLM fair use or not? Where does fair use start end? Academic use? Commercial use?
Who actually owns the stuff an AI produces? What about the stuff there was no consent aquirred to be used as training data? What about AI routinely ignoring robots.txt and sneaking in through spoofing? Is that enough to claim unauthorized use?
But how can you claim that since the models are closed you cannot know what they ingested.
The copyright issues around are numerous and they do not stop at scraping
edit: Courts have also long since decided that snippets and indexing of search engines is fair use indeed. Ask google and its cases and the likes of Axel Springer. (They did lose with Google News)