Skip to main content

Create an AMP crawler

  • Updated

Create automated tests that crawl your website from a specified start URL. AMP saves the crawler test reports to your asset. Run AMP crawler tests on-demand for immediate results or set a recurring schedule and monitor your compliance over time.

On this page:

Create a test

To create a test:

  1. In AMP, select Assets.
  2. Find the asset you want to create the test for.
  3. Select the asset name.
  4. Under Asset Navigation, choose Tests.Asset tests location.
  5. Select Create New Test.

Configure the test

You can configure your test in one of two ways: 

  • Classic: Crawls your website from the start URL.
  • CSV Upload: Crawls your website from each start URL listed in the CSV. If you want to scan a list of pages instead, create an AMP test with a list of pages.

Classic test

To set up a classic test:

  1. Enter and select the test configuration requirements.
  2. Optionally, set up:
  3. Select Submit.

CSV upload

To set up a test by uploading a CSV:

  1. Select CSV upload.
  2. If you haven't:
    1. Download the template for the CSV under Publish Document Inventory.
    2. Fill in the details.
    3. Save the file to your computer
  3. Choose Create one test for each URL.
  4. Enter and select the test configuration requirements.
  5. Optionally, set up:
  6. Select Submit.

Test configuration details

Test configuration requirements

The table below describes the classic test configuration requirements.

Field Description
Test Name Enter the name of the test. Every time the test is run, it will auto-populate the date of the test to differentiate the test reports.
Report Owner Select the user you want to own the reports from this test.
Start URL Enter the start page of the crawl. The crawler tests any pages after the start page.
Viewport Size Select the viewport size. Default viewport size is 1024x768 but you can change it to test the accessibility of responsive websites.
Max Page Count Select the maximum number of pages to test. Default is 10 pages.
Publish Document Inventory Select this option if you want files published to the Document Inventory for future testing.
Document Upload your completed CSV file. This option is only available for CSV upload method. 

Test scheduling (optional)

The table below describes the classic test scheduling options.

Field Description
Date of first report Select the date of the first test scan.
Report start time Select the time of the test scan. Each recurring scan will be run at this time.
Recurrence Select the recurrence.

Macros & authentication (if required)

The table below describes the Macros & Authentication options.

Field Description
Upload Selenium Script Upload a Selenium Script to authenticate log in to the environment you're testing before the test is run. AMP accepts JSON, HTML, and SIDE
Select Macro Select Macros to log in to pages or activate user interface components during the test when the spider lands on the page.
HTTP Basic Authentication Enter authentication details for the spider to use when you run the test if your webpage uses basic authentication.

Advanced test configurations (optional)

The table below describes the advanced test configuration options.

Field Description
Ignore Iframes Ignore iframes and their violations during the test.
Apply Scope To IFrames

Only tests iframes if they're in scope.

For example, if the scope is www.levelaccess.com, and the homepage includes one iframe from the website and one from the LinkedIn feed, this option only tests the iframe from the Level Access website.
Ignore URL Fragments Ignores URL fragments from link harvesting to reduce the number of URLs being tested. URL fragments specify a location within a page using #.
Ignore Query String

Ignores URL query strings during link harvesting to reduce the number of duplicate URLs.

For example, the words you enter into a search bar on a webpage are in the URL as a query string.
Maximum Depth

The number of sub-levels you'd like to test. Depth aligns with the number of backslashes in a URL.

For example, https://www.levelaccess.com/earesources/improved-efficiency-reduced-tech-debt-why-and-how-platform-provider-socure-adopted-agile-accessibility has a depth of two. We recommend a depth of at least one.
Max. Argument Count

The number of unique argument pages to test for a given base URL.

For example, the URL https://www.acme.com/prod.php&product_id=123 has one argument, which is product_id=123. If you set the Max Argument Count to three, AMP will only test the first three pages that it finds with the argument 'product_id'. After three, it won't test the additional pages.
Browser Timeout The time limit, in seconds, for the spider to establish a connection with a page.
Negative Filters

Defines a regular expression (using standard PHP syntax) that specifies pages that you don't want to test.

For example, .*cgi-bin.*, which ignores all URLs that contain "cgi-bin". Negative Filters are | (pipe) delimited.
Positive Filters

Defines regular expression (using standard PHP syntax) that specifies pages or hosts that you want to test, essentially extending the Page Restriction setup. Positive Filters are | (pipe) delimited.

For example, .*www\.levelaccess\.com/contact.php.* or .*www\.levelaccess\.com.*.
XPath Exclusions

Excludes a common section of your website from testing. For example, a specific div element. Format your XPath exclusions like this:

/html/body/div[@id="RandomAds"], and separate additional XPaths using commas.

Note: the XPath Exclusion field supports full XPath Syntax.

For example, the not() XPath expression, as in:

//*[not(ancestor-or-self::main)]

Use this flexibility to narrow in on the specific portion of the page you'd like to test in your spiders.
User Agent Test using a mobile user agent or Firefox. If you leave this field empty, AMP tests using Chrome.
Scope

The basic restriction on the type of pages you want to test.

Path Restriction only tests pages that are present at or below the path of the Start Location. For example, if the Start Location is http://www.levelaccess.com/contact.php, you'll only test pages in the contact directory or its sub-directories on the www.levelaccess.com server.
Restrict to Host only tests pages that are present on the Start Location's host. For example, if the Start Location is http://www.levelaccess.com/contact.php, you'll test any pages on www.levelaccess.com.
Restrict to Domain only tests pages that are present on the Start Location's domain. For example, if the Start Location is http://www.levelaccess.com/contact.php, you'll test pages on www.levelaccess.com, amp.levelaccess.com, or any other server in the levelaccess.com domain.

Run the test

After you create the test the test dashboard will load. To run the test, select Run Test under Test Actions.

Related articles

Comments

0 comments

Article is closed for comments.