Discussions
simple issue: entire basic html website but behind a password page - how?
How can i login the site with 80 legs and let it crawl after it authenticated?
Posted by John Silver over 2 years ago
Inconsistent crawling for same set of data
Hi,
We were scraping one of the websites by making our own app and URL list. Now, some weird behavior we are observing i.e. for the same set of URLs we are getting different outputs. It is inconsistent.
Is it because all the 80legs IPs are blacklisted by that particular website?
Kindly reply to the above issue.
Thanks!
Posted by Romil Shah almost 3 years ago
Crawl limit is lower than my plan
Trying to start a new crawl this morning, but for some reason I am being limited to 10,000 URLs instead of the 100,000 URLs per my pricing plan. (crawls were working normally yesterday)
Posted by Fred Harrell almost 3 years ago
HELP !!!
Hi,
I'm trying the product and created a crawl with the following link:
http://www.bing.es/search?q=Agile+Coach+en+Madrid&count=100&first=600
I also placed to extract emails and go 10 levels inside, but nothing happens (returns 0 and says completed). What am I doing wrong?
You can check all the craws in my account and you'll see that none of them work.
Posted by erich almost 3 years ago
I'm trying to create a crawl and keep getting an error.
I've attempted to create a crawl and keep getting errors. I'm trying to scrape for psychic and intuitive guide advisors on sites like yelp.com and etsy.com. I'd appreciate your help with getting this set up.
Posted by Melanie deLeon about 3 years ago
Amazon scraping
Is it possible to crawl Amazon and get buy box prices and other info using a list of ASINs?
If possible, how?
Posted by T Nakamura about 3 years ago
Throttling requests?
Is it possible to throttle the request rate?
Posted by Pat about 3 years ago
crawl not starting
I have a long list of urls and crawl status says "STARTED" but # urls crawled stays at 0
Posted by d about 3 years ago
Not crawling every page
So I have found that when crawling some URL's it doesn't crawl all the pages?
For example anderstore.com only returns 2 pages but there are many more.
The crawl level is at 5.
Also this URL www.sedgemoorfire.co.uk doesn't return any image links when on the default image crawler.
Any ideas on how to fix this?
Posted by James over 3 years ago
why am i getting error?
my dashboard says
There has been an error
No payment information on file. Visit the 80legs portal to reactive your account.
and i cant add any urls to url list or execute any jobs.
Posted by megan over 3 years ago