Skip to main content

AMP compliance scores

  • Updated

AMP compliance scores are a key part of the AMP Prioritization Model which is designed to provide a value-weighted list of best practices defining those that provide the most benefit, or "bang for the buck", if fixed first.

On this page:

What's a compliance score?

AMP Compliance scores are calculated from the ground up. Calculating first the compliance for each paragraph (rule) of an accessibility standard. Once the compliance for each rule is calculated, AMP then calculates the compliance score for the accessibility standard. Finally, AMP calculates the compliance score for a report based on all the individual accessibility standards tested in the report.

New in the Spring 2014 update, AMP now also offers projected compliance rates (see AMP Compliance Scores FAQ) to give you a better idea of the actual accessibility of tested content by estimating the results of manual testing before the manual testing is even done. This document will explain how we calculate all of these compliance scores that allow you to test monitor and remediate your technology products.

Calculating compliance of accessibility standard rules

The first level of compliance calculations done in AMP are calculating the compliance of each rule of an accessibility standard. To calculate the compliance rate of any given accessibility paragraph (rule) AMP uses the following formula:

ParagraphCompliance = (floor((ModulesWithNoViolation/TotalModuleCount)*TotalParagraphRuleScore)))/TotalParagraphRuleScore

floor: indicates that the result will be rounded down
ModulesWithNoViolation: The number of modules in the report that have no violations of the current paragraph of the current accessibility standard being tested.
TotalModuleCount: The total number of modules in the report that are being tested.
TotalParagraphRuleScore: The total rule score is the proprietary score given to each accessibility rule the accessibility experts at Level Access.

Now that we have calculated a rule compliance we can look at calculating a standard compliance.

Calculating compliance of an accessibility standard

Having calculated the compliance scores for each rule in a standard, AMP can then calculate the compliance for the standard. This is calculated using the following formula:

standardCompliance = floor(sumOfAllParagraphRuleScores/totalPossibleStandardRuleScore)

floor: indicates that the result will be rounded down
sumOfAllParagraphRuleScores: This is the sum of all the rule scores for a standard found via the above equation for 'paragraphCompliance'
sumOfTotalPossibleParagraphRuleScores: This is the total possible score for an accessibility standard. it is derived from the sum of each rule's potential score.

Here is an example of this: WCAG A has a possible total 250 points. This report scored 211 points. 211/250 = 84.4 rounded down to 84%

Calculating overall report compliance

Calculating the overall report compliance is handled the same way we calculate the compliance for an accessibility standard, which is by finding quotient of the total calculated score of every accessibility standard divided over the sum of the total possible score of each standard. This Formula looks like this:

overallReportCompliance = floor(sumOfAllStandardRuleScores/sumOfTotalPossibleStandardRuleScores)

floor: indicates that the result will be rounded down
sumOfAllStandardRuleScores: This is the sum of all the standard rule scores for all active standards in the organization. It is derived by summing each active standard's 'sumOfAllPararaphRuleScores'.
sumOfTotalPossibleStandardRulesScores: This is the sum of all total possible scores of active accessibility standards across the organization.

Here is an example of this: The overall points scored on this report across all three standards was 404 out of a possible 480. Therefore, 404/480 = 84.1666666666 rounded to 84%

Calculating asset compliance

Calculating the compliance for a asset is handled by three rules:

  • Include latest report generated from a test
  • Exclude all reports with 0 modules
  • Include all reports not generated from a test

What this means is that to provide a useful asset compliance metric we make sure that if multiple reports are generated from the same test we only factor the compliance score from the most recent test. Because our users use AMP Tests to create primary and regression audits, factoring only the most recent report from a test allows us to only include the most recent regression cycle so that your asset compliance score remains accurate and up to date. By excluding any reports with 0 modules we make sure no failed tests are factored into this score. Lastly we make sure that for reports that are not generated via an AMP Test (e.g. I manually create a report, reports based on toolbar testing, etc) include them all by default as long as they have at least 1 module. This allows to make sure that any manual tests that aren't being run in a regression fashion are going to be factored into your score to again make sure we are providing you with the most accurate metrics possible.

Visual representation of compliance scores

Compliance Scores are presented on the Report Dashboard, represented with a percentage value and a color coded background.  The color codes for compliance scores are:

  • Red = 0% - 69%
  • Yellow = 70% - 99%
  • Green = 100%

An example of the color coded background can be seen below visually in the screenshot:

Compliance score chart.

 

Frequently asked questions

Q: Is the compliance score a recognized industry standard measure?

A: There is no industry standard measure. However, the compliance calculation is a simple formula that calculates compliance based on the violations against the mapped standards. It provides an accurate level of conformance but it doesn’t tell the whole story about risk or health of the site.

Specifically it doesn’t weigh the modules based on frequency of use. It also scores based on all standards that are mapped via best practices in the report or organization – so if a standard isn’t applicable you get credit for passing it. This tends to make compliance scores appear higher.

Q: What score is “good enough?”

A: 100% is the ideal level and is the only true level of conformance. A rule of thumb would be that 97-98% or above is the goal as there will always be bugs in software."

Q: How can my score for WCAG 2.0 AA be higher than my score for WCAG 2.0 A?  Isn't WCAG 2.0 AA compliance reliant on WCAG 2.0 A?

A: The reason why this occurs is because each standard (Level A and AA) are calculated separately. When we say that you have 85% for WCAG AA, we mean that you have 85% for all success criteria that are indicated at the AA Level. While you may only be 78% conformant to the success criteria that are mapped to Level A.

In order to conform with WCAG Level AA you must conform to all level A success criteria and all Level AA success criteria. Thus, even if you met all of the Level AA success criteria but failed to meet all of Level A then you are not conformant. In order to be fully conformant you have to meet all success criteria at each level. We separate these standards out as some people want to measure their level A conformance separately. Level A is the minimum level of accessibility. Level AA builds on that minimum level and thus having separate metrics is important to understand where the minimum level is in regards to conformance.

For additional FAQs regarding Compliance Scores, visit our Compliance Scores FAQ Section in our documentation.

 

Related articles

Comments

0 comments

Article is closed for comments.