Discussions

How many crawls per month is actually possible?

almost 5 years ago by Courtney Rogers

What are the top users actually able to crawl? Billions of urls? Is the structure of 80legs able to facilitate such feats?

﻿