Reddit has updated its robots.txt to block all web crawlers (stackdiary.com)
Web Crawling, Robots.txt, Reddit, Social Media
19 points by skilled 141 days ago | 15 comments