Discussions

Hi There, The output from the crawling consistently shows

expected result of processDocument to be a non-circular object or array of non-circularobjects

Testing the app using the test framework returns the results we are looking for.

The processDocument script is capturing all external links and storing them in array, then converting this array to a JSON and adding it to the return object.

app.processDocument = function (html, url, headers, status, $) {
    const $html = this.parseHtml(html, $);
    const links = [];
    const object = new Object();

    const r = /:\/\/(.[^/]+)/;
    const urlDomain = url.match(r)[1];
    const normalizedUrlDomain = urlDomain.toLowerCase();

    // gets all links in the html document
    $html.find('a').each(function (i, obj) {
        const link = app.makeLink(url, $(this).attr('href'));

        if (link) {
            const linkDomain = link.match(r)[1];

            if (linkDomain.toLowerCase() !== normalizedUrlDomain) {
                if (!links.includes(linkDomain)) {
                    links.push(linkDomain);
                }
            }
        }
    });

    object['list'] = JSON.stringify(links);

    return object;
}

Any help on how to recode this so that we can get an array of external links as opposed to this error message would be great.

Scott