AMP crawler advanced configuration

Choose a basic restriction on the type of pages you want to test:

• Path Restriction only tests pages that are present at or below the path of the Start Location. For example, if the Start Location is http://www.levelaccess.com/contact.php, you'll only test pages in the contact directory or its sub-directories on the www.levelaccess.com server.
• Restrict to Host only tests pages that are present on the Start Location's host. For example, if the Start Location is http://www.levelaccess.com/contact.php, you'll test any pages on www.levelaccess.com.
• Restrict to Domain only tests pages that are present on the Start Location's domain. For example, if the Start Location is http://www.levelaccess.com/contact.php, you'll test pages on www.levelaccess.com, amp.levelaccess.com, or any other server in the levelaccess.com domain.

Choose to ignore iframes during testing so that no violations are reported.

Choose to test iframes only if they're in scope. For example, the scope is www.levelaccess.com, and our homepage includes one iframe from another page on our site and one from our Linkedin feed. Choosing this option would only test the iframe that came from another page on our site and skip the one that came from Linkedin.

Choose this option to ignore the URL fragment during link harvesting to reduce the number of duplicates urls being tested. 

URL fragments specify a location within a page and always contain the # character.

Ignore URL query strings during link harvesting to reduce the number of duplicate URLs.

An example of a query string is the words you enter into a search bar on a webpage. Those words appear in the URL as a query string.

Choose a depth to indicate how many sub-levels of your website you'd like to test.

Depth aligns with the number of backslashes in a URL. For example, https://www.levelaccess.com/earesources/improved-efficiency-reduced-tech-debt-why-and-how-platform-provider-socure-adopted-agile-accessibility has a depth of two. We recommend a depth of at least one.

Choose the number of unique argument pages to test for a given base URL. For example, the URL https://www.acme.com/prod.php&product_id=123 has one argument, which is product_id=123. If you set the Max Argument Count to three, AMP will only test the first three pages that it finds with the argument 'product_id'. After three, it won't test the additional pages.

Sample: Maximum Argument Count = 3

www.acme.com/prod.php&product_id=123 (will test)

www.acme.com/prod.php&product_id=124 (will test)

www.acme.com/prod.php&product_id=156 (will test)

www.acme.com/prod.php&product_id=876 (won't test, as its the fourth argument of the same kind and the maximum argument count is set to 3).

However, www.acme.com/prod.php&product_id=743&type=onsale would test based on the addition of a new argument in the URL.

Defines a regular expression (using standard PHP syntax) that specifies pages that you don't want to test.

An example of this is .*cgi-bin.*, which ignores all URLs that contain "cgi-bin". Negative Filters are | (pipe) delimited.

Defines regular expression (using standard PHP syntax) that specifies pages or hosts that you want to test, essentially extending the Page Restriction setup. Positive Filters are | (pipe) delimited.

Examples this are:

• .*www\.levelaccess\.com/contact.php.*
• .*www\.levelaccess\.com.*
• .*levelaccess\.com.*

Specify a common section of the site to exclude from testing. For example, a specific div element.

Format XPath Exclusions like this /htm/body/div[@id="RandomAds"], and separate additional XPaths using commas.

Note: the XPath Exclusion field supports full XPath Syntax, including, for example, the not() XPath expression, as in:

//*[not(ancestor-or-self::main)]

or

//*[not(ancestor-or-self::footer)],//*[not(ancestor-or-self::header)]

You can utilize this flexibility to narrow in on the specific portion of the page you'd like to test in your spiders.

Add a user agent string to test using a mobile user agent or Firefox. If you leave this field empty, AMP tests using Chrome.