A few possibilities,
- brave has state in the past they use the googlebot user agent, if all reddit does is check the useragnet, it won’t block brave. This does however mean brave is violating the robots.txt file.
- I saw some mentions of google fall back, I don’t know if they still do it, but that could be another possibility.
- Brave ignores robots.txt files
- Brave paid for access
No matter the reason, well behaving crawlers will no longer crawl reddit, Everything is disallowed in the robots.txt
User-agent: *
Disallow: /
Reddit believes in
an openpay to access internet, but not themisuse ofpublic contentour content we didn’t make.