Discussions
Inconsistent crawling for same set of data
over 2 years ago by Romil Shah
Hi,
We were scraping one of the websites by making our own app and URL list. Now, some weird behavior we are observing i.e. for the same set of URLs we are getting different outputs. It is inconsistent.
Is it because all the 80legs IPs are blacklisted by that particular website?
Kindly reply to the above issue.
Thanks!