How to crawl big websites with no sitemap? (ycombinator.com)