Discussions

Ask a Question
Back to All

Inconsistent crawling for same set of data

Hi,
We were scraping one of the websites by making our own app and URL list. Now, some weird behavior we are observing i.e. for the same set of URLs we are getting different outputs. It is inconsistent.
Is it because all the 80legs IPs are blacklisted by that particular website?

Kindly reply to the above issue.

Thanks!