[Home - 0x79 Welcome! check out my blog: blog.0x79.com Crawler I am running a crawler from this site for a university project. I try to follow any robots.txt rules and keep a 2-5 second interval between each request. You can checkout the source-code of the crawler here If you do ...]