Skip to main content

Modify headers

  • Updated

You can override the default values of the standard headers we send if you like or can supply additional headers we don’t normally send.

On this page:

What are headers?

HTTP headers are used to pass additional information in the requests our system makes to your web servers. Headers are formatted as a name set to a value. There are many standard headers such as User-Agent, which is set to a string of characters that identifies the browser type, version and operating system. You can override the default values of the standard headers we send. 

Many sites have various forms of bot protection that can thwart our system from scanning your site. One common technique to resolve this is to send a custom header as a secret handshake. Your web server would need to notice this custom header and then respond by stopping its normal protections. For example, you could add a made up custom header “ignoreCaptcha=true”.

Your web developer could then modify the web server code to not present a CAPTCHA challenge when it sees this custom header. Your web developers may already have some custom header responses defined to ease testing by QA or automated tooling.

Learn more about HTTP headers.

Here are some custom headers your development team might want to implement support for. Your site implementation would respond to the custom headers as a signal to turn off behaviors that impede our systems scanning your pages:

Use case Solution Sample header Sample value
Your scans are failing because passing a Captcha test is required to access your website. Add a header to your website that allows the scanner to bypass the captcha requirement. ignoreCaptcha true

You want to crawl your website and the scan fails because your website has security mechanisms that prevent unauthorized crawling.

Add a header to your website that tells the mechanism that the scanner is safe and allowed to crawl.

 Allowcrawl true

Your site has pop-ups that must be closed before you can access the content on the page.

 Add a header to your website to never display pop-ups when it receives a certain string in the header of the request. Closepopups true

Your site's elements vary based on the end user's browser and you want to scan your website as a Chrome user.

 Use a user-agent string to run a scan on the Chrome version of your website. User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 

If you’re not sure whether you need to use the Modify headers option, contact your IT department.

Add a header and value pair

To add a header and value pair to an scan:

  1. Go to Digital assets.
  2. Select the digital asset you'd like to scan.
  3. Choose Scans.
  4. Select Run scan.
  5. Choose the Advanced tab.
  6. Fill out the required fields.
  7. Select the Modify headers option.
  8. Enter a header and value.
  9. (Optional) To add additional pairs of headers and values, select Add pair.
  10. Select Run scan.

You can save header and value pairs by saving scan options.

Delete a header and value pair

To remove a value pair:

  • Deselect the Modify header option.

To remove a value pair when you have multiple pairs:

  • Select Delete.

Related to

Related articles

Comments

0 comments

Article is closed for comments.