Running crawls with PHP

For this guide, we're going to assume you're interested in scraping keywords from a specific list of websites you're interested in.

1. Open a code editor

If you want to use PHP to access the 80legs API, we're assuming you'll be using a standard code editor to write your PHP code. Open your code editor to get started.

2. Get your API token

The next thing you'll need is your API token. The API token lets you authenticate with 80legs API and tells it who you are, what you have access to, and so on. Without it, you can't use the API.

To get your API token, go the 80legs Web Portal (https://portal.80legs.com), login, and click on your account name and the top-right. From there, you'll see a link to the "My Account" page, which will take you to a page showing your token. Your API token will be a long string of letters and numbers. Copy the API token or store it somewhere you can easily reference.

📘

For the rest of this document, we'll use AAAXXXXXXXXXXXX as a substitute example for your actual API token when showing example API calls.

3. Upload your URL list

Before we can create our web crawl, we need to create a URL list. A URL list is one or more URLs from which your crawl will start. Without the URL list, a crawl won't know where to start.

Write the following code in your code editor (replace the dummy API token with your real API token):

<?php

// Set your API parameters here.
$APIToken = 'AAAXXXXXXXXXXXX';
$urllist_name = 'urlList1';
  
$url = 'https://' . $APIToken . ':@api.80legs.com/v2/urllists/' . $urllist_name;

$request_body = array(
	'https://www.80legs.com',
  'https://www.datafiniti.co'
);

$options = array(
	'http' => array (
		'header'  => 'Content-Type: application/octet-stream',
		'method'  => 'PUT',
		'content' => json_encode($request_body)
	)
);

$context  = stream_context_create($options);
$result = file_get_contents($url, false, $context);
if ($result === FALSE) {
	echo "Error.";
} else {
	var_dump($result);
}

?>

In this example, we're creating a URL list with just https://www.80legs.com and https://www.datafiniti.co. Any crawl using this URL list will start crawling from these two URLs.

You should get a response similar to this (although it may not look as pretty in your terminal):

{
  "created": "urlList1"
}

4. Upload your 80app

The next thing we'll need to do is upload an 80app. An 80app is a small piece of code that runs every time your crawler requests a URL and does the work of generating links to crawl and scraping data from the web page.

You can read more about 80apps here. You can also view sample 80app code here. For now, we'll just use the code from the KeywordCollector 80app, since we're interested in scraping keywords for this example. Copy the code and save it your local system as keywordCollectorjs.

Write the following code in your code editor (replace the dummy API token with your real API token and /path/to/keywordCollector.js with the actual path to this file on your local system):

<?php

// Set your API parameters here.
$APIToken = 'AAAXXXXXXXXXXXX';
$app_name = 'keywordCollector.js';
  
$url = 'https://' . $APIToken . ':@api.80legs.com/v2/apps/' . $app_name;

$path_to_app = '/path/to/keywordCollector.js';

$options = array(
	'http' => array (
		'header'  => 'Content-Type: application/octet-stream',
		'method'  => 'PUT',
		'content' => file_get_contents($path_to_app)
	)
);

$context  = stream_context_create($options);
$result = file_get_contents($url, false, $context);
if ($result === FALSE) {
	echo "Error.";
} else {
	var_dump($result);
}

?>

You should get a response similar to this (although it may not look as pretty in your terminal):

{
  "created": "keywordCollector.js"
}

5. Configure and run your crawl

Now that we've created a URL list and an 80app, we're ready to run our web crawl!

Write the following code in your code editor (replace the dummy API token with your real API token):

<?php

// Set your API parameters here.
$APIToken = 'AAAXXXXXXXXXXXX';
$crawl_name = 'testCrawl1';
  
$url = 'https://' . $APIToken . ':@api.80legs.com/v2/crawls/' . $crawl_name;

$request_body = array(
  'urllist' => 'urlList1',
	'app' => 'keywordCollector.js',
  'max_depth' => 10,
  'max_urls' => 1000
);

$options = array(
	'http' => array (
		'header'  => 'Content-Type: application/json',
		'method'  => 'PUT',
		'content' => json_encode($request_body)
	)
);

$context  = stream_context_create($options);
$result = file_get_contents($url, false, $context);
if ($result === FALSE) {
	echo "Error.";
} else {
	var_dump($result);
}

?>

You should get a response similar to this (although it may not look as pretty in your terminal):

{
    "id": 123456,
    "name": "testCrawl1",
    "user": "abcdefghijklmnopqrstuvwxyz012345",
    "app": "keywordCollector.js",
    "urllist": "urlList1",
    "max_depth": 10,
    "max_urls": 1000,
    "status": "STARTED",
    "depth": 0,
    "urls_crawled": 0,
    "date_created": "2018-7-25 19:0:54",
    "date_completed": "",
    "date_started": "2018-7-25 19:0:54"
}

Let's break down each of the parameters we sent in our request:

Now let's dive through the response the API returned:

6. Check on crawl status

As mentioned, there is a status field in the response body above. This field shows us the crawl has started, which means it's running. Web crawls typically do not complete instantaneously, since they need to spend requesting URLs and crawling links. In order to tell if the crawl has finished, we can check on its status on a periodic basis.

Write the following code in your code editor (replace the dummy API token with your real API token):

<?php

// Set your API parameters here.
$APIToken = 'AAAXXXXXXXXXXXX';
$crawl_name = 'testCrawl1';
  
$url = 'https://' . $APIToken . ':@api.80legs.com/v2/crawls/' . $crawl_name;

$result = file_get_contents($url, false);
if ($result === FALSE) {
	echo "Error.";
} else {
	var_dump($result);
}

?>

You'll get another crawl object as your response like this:

{
    "id": 123456,
    "name": "testCrawl1",
    "user": "abcdefghijklmnopqrstuvwxyz012345",
    "app": "keywordCollector",
    "urllist": "urlList1",
    "max_depth": 10,
    "max_urls": 1000,
    "status": "STARTED",
    "depth": 1,
    "urls_crawled": 198,
    "date_created": "2018-7-25 19:0:54",
    "date_completed": "",
    "date_started": "2018-7-25 19:0:54"
}

If you keep sending this request, you should notice depth and urls_crawled gradually increasing. At some point, status will change to COMPLETED. That's how you know the crawl has finished running.

7. Download results

After the crawl finishes, you'll want to download the result files. Result files are logs of all the data scraped during the crawl.

Once you see a status of COMPLETED for your crawl, use the following code to get the results (replace the dummy API token with your real API token):

<?php

// Set your API parameters here.
$APIToken = 'AAAXXXXXXXXXXXX';
$crawl_name = 'testCrawl1';
  
$url = 'https://' . $APIToken . ':@api.80legs.com/v2/results/' . $crawl_name;

$result = file_get_contents($url, false);
if ($result === FALSE) {
	echo "Error.";
} else {
	var_dump($result);
}

?>

You should get a response similar to this (although it may not look as pretty in your terminal):

[
    "http://datafiniti-voltron-results.s3.amazonaws.com/abcdefghijklmnopqrstuvwxyz012345/123456_1.txt?AWSAccessKeyId=AKIAIELL2XADVPVJZ4MA&Signature=P5aPspt%2B%2F0Kr8u1nxU%2FHVVrRgOw%3D&Expires=1530820626"
]

Depending on how many URLs you crawl, and how much data you scrape from each URL, you'll see one or more links to result files in your results response. 80legs will create a results file for every 100 MB of data you scrape, which means result files can post while your crawl is running.

For very large crawls that take more than 7 days to run, we recommend checking your available results on a weekly basis. Result files will expire 7 days after they are created.

To download the result files, you can run code like this:

<?php

// Set your API parameters here.
$APIToken = 'AAAXXXXXXXXXXXX';
$crawl_name = 'testCrawl1';
  
$url = 'https://' . $APIToken . ':@api.80legs.com/v2/results/' . $crawl_name;

$result = file_get_contents($url, false);
if ($result === FALSE) {
	echo "Error.";
} else {
	$result_list = json_decode($result,true);
  for ($i = 0; $i < count($result_list); $i++) {
			$result_link = $result_list[$i];
			$filename = $crawl_name . '_' . $i . '.txt';
			file_put_contents($filename, fopen($result_link, 'r'));
			echo 'File ' . ($i+1) . ' out of ' . count($result_list) . ' saved: ' . $filename . '\n';
	}
}

?>

8. Process the results

After you've download the result files, you'll want to process them so you can make use of the data. A result file will have a structure similar to this:

[
  {
    "url": "https://www.80legs.com",
    "result": "...."
  },
  {
    "url": "https://www.datafiniti.co",
    "result": "...."
  },
  ...
]

Note that the file is a large JSON object. Specifically, it's an array of objects, where each object consists of a url field and a result field. The result field will contain a string related to the data you've scraped, which, if you remember, is determined by your 80app.

In order to process these results files, you can use code similar to this:

<?php

// Set the location of your file here
$file = "xxxx_x.txt";

$records = array();

$handle = fopen($file, "r");
if ($handle) {
	while (($line = fgets($handle)) !== false) {
		$records[] = json_decode(($line));
	}
} else {
	echo "Error opening file.";
}

foreach($records as $record) {
	// Edit these lines to do more with the data.
	print_r($record);
}

?>

You can edit the code in the foreach loop above to do whatever you'd like with the data, such as store the data in a database, write it out to your console, etc.

📘

For this guide, we have created separate code files or blocks for each step of the crawl creation process. We've done this so you can understand the process better. In practice, it's probably best to combine the code into a single application to improve maintainability and usability.