5 things you should know before buying accessibility audit and accreditation services

Note on February 2021 update:

This blog has always been popular since I wrote it back in 2013. The original version was a little about accessibility audits, and a lot about accreditation.

But a lot has changed since then. So I felt like it was time for an update.

There are more and more companies out there in 2021 that could do you an accessibility audit: some with a great pedigree, others are new to the stage. Some are expensive. Others seem really cheap. So, do you get what you pay for? And what is worth paying for?

Here’s what you need to know in 2021, all based around the concept of the “total cost of ownership” of accessibility for you, in the situation you find yourself in:

1. Why do you want an audit?
2. Understanding you get what you pay for in an audit
3. It’s not the cost of the audit that matters, it’s the round-trip “cost of ownership” that does
4. It’s all about the journey, not the false-start
5. What sort of accreditation is actually valuable in 2021?

1. Why do you want an audit?

The first thing to think about is why you want an audit.

In our experience, there are a few key reasons:

• You’re looking to assess your website or app’s accessibility to find any weaknesses so you can fix them all:
  • This wins you peace of mind on compliance, and also real progress in making your site or app accessible to more people, which can deliver the benefits of more customers and lower customer-support costs.
  • You need a comprehensive audit service that will support your product’s accessibility progress, rather than just paying for a minimal report.
• You want to assess the level of accessibility risk in your site or app, and do what’s necessary to meet minimum requirements:
  • It could be that you are a software-as-a-service company with a new client asking about your service’s accessibility, so you want to show some movement towards accessibility compliance, but you don’t think it’s a business priority to do too much yet. You need a VPAT and/or Accessibility Statement to sell your product to them, based on enough of an accessibility audit to be able to fill out each to a minimal level of completeness.
  • If you’re not going to fix more than the top 5 or 10 easiest and cheapest things to fix, you need a light-touch audit that focuses on your key user journeys, and on quick fixes and workarounds for the top issues found.
  • But, be aware, that accessibility statements that are not kept up to date, and say you’ll fix things at some vague point, get less and less effective as a risk mitigator over time.
• Are you looking to assess the level of accessibility, but are not actually planning to do anything about it?
  • It could be that you need to baseline the accessibility of a product that is old and few people use, or a product that you’re going to totally rebuild for a v2. If you aren’t going to fix things, or re-use the design or code used on the site, you may have little to learn from knowing where its accessibility was strong or weak.
  • A minimal report is all you need for the record.

2. Understanding you get what you pay for in an audit

Here I’m going to let you into a bit of a secret.

The cost of an audit, like any testing, depends on how much your supplier tests, how deeply they test, how much documentation you want them to write, and how much quality you want them to bring to each of those aspects of the audit:

• Finding the issues:
  • The cost of this depends on the number of pages you want audited, and the complexity of those pages. This is why audit suppliers will need to review your site to scope out the number of pages they need to audit. The only way to do an audit without this sort of scoping is to do something like our Live Audits, which are fixed length, based on your site or app’s 3 key user journeys.
  • It also depends on how many devices, browsers and assistive technologies you’re testing against, which depends on the product and its audiences. This could be simple and inexpensive, for example if you are testing a intranet where you know all your staff will be using it on a PC running Windows 10, using only IE11, and the Jaws screen reader (because that’s what you provide to staff who are blind); or testing a mobile app which is only available on iOS. Or it could be complex and very expensive, if you’re testing a public website, which people could use on laptop (PC and Mac, with all the different browsers and multiple assistive technologies on each of those) or mobile (iOS and Android, with the different assistive technologies on each of those).
  • Finally, the cost is also impacted by when you’re happy for the auditor to stop looking for examples of an accessibility issue (defect) when they’ve already found many. For example, at Hassell Inclusion, we often have 4 or 5 types of fail under each WCAG success criterion (checkpoint), each requiring a different type of fix. We document up to 3 or 4 instances of each one. Then, if you are going to fix issues, you can look for any others yourself.
• Analysis of the fails:
  • The cost of this depends on the number of fails found, and whether standard ways of fixing them would work for your site or app (which takes very little time) or some issues are new ones which only show up with some assistive technologies or because you’ve used unusual coding patterns (which can take lots of time to understand).
  • This is the first hint that the real cost of your audit actually depends on how good or bad the accessibility of your site or app is.
• Write-up of the fails into your audit report:
  • The cost of the audit massively depends on how many issues the auditor needs to document. Even with a great report template to speed this up, this again depends on the number of issues they find, and the amount of information they provide in the report to help you fix things.
  • If an audit finds very little wrong, it could take a few hours to write up a short report telling you that all is well. If your site or app fails most of WCAG, the report could go on for hundreds of pages, which could take weeks to write.

All accessibility auditors know that these are the factors that will dictate the cost of the audit. They’ll be able to control the number of pages to test (the scope) when giving you a quote for the audit. But they won’t be able to predict the number of fails they find. So they won’t know how much the audit will actually cost.

However, you want a fixed quote. So they’ll estimate the number of days they think it will take them. And then they will fit the audit into that number of days, under-reporting any difficulties they don’t have time to look into. So it’s useful to have discussed with your auditor if they should contact you to keep you informed during the audit of how its going.

There are three things to understand out of this:

1. Auditors who have been auditing for a long time, are likely to be quicker at finding issues than those who are new to it, and more accurate at finding all issues.
2. Organisations which have been auditing for a long time are likely to have a huge library of fixes that the auditor can quickly put into your report to suggest a fix for issues found in your site or app. Those who are new to the game would need to cut and paste from WCAG (which can be difficult to understand), or write their own (which might take lots of time).
3. That goes double for any complex issues found that only occur in some assistive technologies (especially less known ones like Claroread, for people with dyslexia) or are not part of a “standard” fix. The best auditors, who have been auditing many types of product for years, will either have these fixes in their database from previous experience or can use that experience to find fixes for totally new issues quickly. They may ask you questions about the product’s target users, the systems/suppliers used to create the product, and how easily you can make changes, and use that information in making recommendations. If they have experience of how people with disabilities use assistive technologies in real life, they may also be able to suggest workarounds to solve problems at minimal cost, where fixes are difficult to make. Those new to auditing may really struggle.

These are the reasons we think it’s a good idea to find out:

• how long the organisation who is pitching you an audit has been auditing for, and
• how long the specific auditor(s) who will be doing your audit have been auditing for.

While an experienced auditor, working for an experienced accessibility company, is likely to command a higher day-rate, they will also get you a much better quality report, in fewer days.

You really do get what you pay for. To use a healthcare analogy:

• Cheap healthcare:
  Patient: Doctor, my back hurts
  Doctor: Here are some painkillers – bye

• Expensive healthcare:
  Patient: Doctor my back hurts
  Doctor: How long has it been hurting for? Maybe some physio would help? Let’s look to see what’s caused it… Maybe get a better chair at work to make sure it doesn’t come back?

So, if you care about being able to fix things found in the audit, it’s worth buying quality.

If you aren’t going to make fixes, you may be able to get away with cheap.

Which brings me to my key point…

3. It’s not the cost of the audit that matters, it’s the round-trip “cost of ownership” that does

I’m guessing that you, like me, have bought many printers in your time. They rarely last more than a few years, and seem to have been made to become obsolete. Which is the exact point.

The key thing when you’re buying a printer, if you’ve read any good reviews, is: the total cost of ownership.

The cost of the printer to you isn’t that initial loss-leader cost – it’s that cost, plus the cost of running that printer until you replace it. And that’s dependent on how many pages (of black and white or colour printing) that you expect to do each year, and how much the toner/drum/ink cartridges will cost to do that printing.

Some printers (usually inkjets) are very cheap to begin with, then sting you in the cost of the ink. Whereas others (usually laser printers) initially seem expensive, but then the toner and drums last for so much longer that the total running cost per page is much cheaper.

It’s the same with accessibility audits.

At Hassell Inclusion we believe in minimising the total cost of accessibility, not minimising the cost of an audit:

• Spending less money on an audit to find less of the problems in your product isn’t good sense if the goal is to fix things.
• Spending less money on an audit is good sense when you found all of the issues that you could find yourself beforehand because you’ve already embedded accessibility testing in your process. If you have done this, you should expect lower audit costs from your supplier (see our case study for how we did this for Scope, to award-winning effect).

However, if you still only have budget for a cheaper audit, we’d suggest that you concentrate the audit on the key user journey on your site, via our Live Audits, rather than going to an agency that appears cheaper, and say they can look at the whole of your website, but actually don’t have time to be able to document all of the issues that they find.

Why? Because a Live Audit highlights key issues, helps you add them into your bug tracking system so you can find them on other pages, and most importantly means you are making progress towards fixing things to make your site more accessible to more people, while teaching you things to stop you making the same mistakes next time.

You need to look at the cost of the audit as the down-payment on the costs of the rest of the journey, not the sole payment.

4. It’s about the journey, not the false-start

An accessibility audit starts you on a journey. And how far you go on that journey depends on what you want to get out of it. Is it just to assess your risk? To mitigate your risk? To help people actually use your service and get some benefits to your business? Or to prove to your customers that they should remain loyal to you because you’re actually going to continue making what you do work?

The choice is yours.

If you just want the piece of paper saying how bad your site’s accessibility is, without wishing to improve it, then spend as little as you can on it, because in reality it’s worth very little.

But if you want to come on the journey to where the real value is, then pick an accessibility company who can take you all the way on that journey, taking time to understand your user journeys and customer touch points, and are able to do it to minimise your total cost of ownership.

We’re not the only company out there who can do that, but I would caution you against choosing companies that “dabble” in accessibility. The demand for accessibility audits in 2021 is high, so we welcome other companies in the accessibility audit market, as long as they provide a good service, with auditors who are well trained and able to audit well.

However, as many organisations found in 2020, choosing companies solely based on price may sound like a good idea, but often turns out to be a false start. We have many clients who’ve come to us having learnt that the hard way – having been delivered audits that were rushed, lacked detail, didn’t take their user-base or customer journeys into account, and just weren’t good enough – and now need help to figure out what to do next.

We’re not a usability company who do a bit of accessibility testing on the side. Or a digital agency who do accessibility audits because we’ve done a few for our own products. We’re a company who can not only give you a great audit, but also help you fix the deep technical issues in your product. We are a company that can help you change the way you do your design and development, so that you don’t have so many problems next time. We’re a company who can train you to test your own accessibility as you create products, so we can charge you less for audits just before launch because we’ll know there will only be minor issues in them to report on.

We’re a company who can help you get good at accessibility. So you get the benefits of accessibility, not just the costs.

If that’s what you want, come to us.

As the CEO of Galaxkey says:

“The advantage of working with Hassell Inclusion was their executives understand the software development process. This made the entire experience seamless for our team to understand and then implement. The last thing any software product company needs is to redesign everything. But the solutions provided by Hassell were easy to implement and super practical.”

5. What sort of accreditation is actually valuable in 2021?

Finally, we come back to Accreditation, which was my key point back in 2013.

A lot of people want to get some sort of accreditation for having done an audit, and fixed a lot of the stuff found. It’s not been easy, so they want some sort of badge, to make all that work worth it.

That’s understandable. But that’s not the thing that you need. Because the accreditation is of one ‘fixed’ product at one point in time – that on one day, that one site/app was compliant.

That’s a major achievement. But, I’m afraid, it’s not the hardest part of accessibility.

Keeping your site or app accessible/compliant is the hardest part. Because it can have two new pages added tomorrow that are both inaccessible and then you’re back in trouble again.

Ideally, what you want is proof not that your website on a single day is accessible, but proof that your company has actually got the processes in place to keep it accessible, all of the time. And to not just keep your website or app accessible, but all the other digital things you own as well (like your social media and your intranet).

That’s why people rarely talk now about the accessibility badges I talked about in 2013. And why they are talking instead about how to get accredited against organisational accessibility standards like ISO 30071-1.

Check out our article on How to make accessibility a habit in your organisation rather than an add-on, and you’ll see what accreditation looks like in 2021.

Need help?

If you need any help in deciding what accessibility audit is right for your organisation, please contact us, and we’ll be delighted to help you.

Want more?

If this blog has been useful, you might like to sign-up for the Hassell Inclusion newsletter to get more insights like this in your email every other week.

The original January 2013 blog:

From WCAG 2.0 AA and Section 508 VPATs to RNIB/AbilityNet Surf Right, DAC and Shaw Trust accreditation, there are a lot of accessibility conformance badges out there.

So how do you know which badge to pick? What is the actual value of these badges to the organisations that buy them, and to the disabled people who use their sites?

These are questions I get regularly asked by clients, so here’s a guide for how to choose the best accessibility accreditation for your website.

1. Why is there more than one accreditation badge?

Yes, there are lots of badges to choose from.

While WCAG 2.0’s level A, AA and AAA badges are close to a de-facto Standard for accreditation, many organisations that carry out accessibility audits soon become aware of the limitations of WCAG’s conformance levels and come up with their own-brand accreditation badges as a value-add for clients, based on their own experience.

In the UK most major organisations that offer accessibility audits have their own badge: DAC’s Accreditation, Shaw Trust’s Accreditation, RNIB See It Right and Surf Right badges and RNIB logo standard with UseAbility.

Accreditation in the United States happens somewhat differently, with VPAT certificates of compliance with Section 508 guidelines often the result of accessibility audits.

And, in December 2012, the Hong Kong Office of the Government Chief Information Officer (OGCIO) launched a Web Accessibility Recognition scheme, which encourages local businesses, NGOs and academia to apply for a free accessibility audit and award of accessibility accreditation badges at new Gold and Silver levels that they have defined.

Unless harmonization occurs in the accessibility audit market, there’s likely to be more, not less, options for accessibility accreditation in the future.

So, when you’re choosing an organisation to audit and accredit your site, ask them the strengths and limitations of the benefits that their badge will bring to you, as well as the costs.  And choose the badge that gives you your preferred balance of costs and benefits.

While costs are easy to assess, benefits are harder. You’ll need to consider two dimensions of benefit:

• value to you – the company owning the website; and
• value to the disabled and elderly users of your website

2. Does the badge bring value to the company owning the website?

Accessibility conformance badges make organisations feel more secure about accessibility.

Chances are you’re interested in a badge because you feel it gives you some sort of external, independent proof and public recognition that you’ve achieved a particular level of accessibility.

If you’re investing in an accessibility audit, you might as well pay a bit more for a badge that summarises the results of that audit in a simple way you can share with your users.

All such badges promise this, other than the WCAG 2.0 conformance badges and Section 508 VPATs whose value has tended to become questioned because, unlike other badges, you can award them to your own products, rather than have to pay someone else to do the testing and accreditation for you.

This sense of security is valuable, especially for your organisation’s reputation management and PR. Displaying the results of having tested against a set of accessibility metrics tends to prove your statements declaring the website’s commitment to accessibility have actually achieved good results that have been independently verified by an reputable auditor.

But part of this sense of security is also potentially misplaced. Having a badge doesn’t necessarily mean your site is usable by disabled people. It won’t guarantee you won’t get sued under discrimination legislation, or save you from having to deal with accessibility complaints from disabled users via email, twitter or Facebook.

So, when you’re choosing an organisation to audit and accredit your site, ask for details on how their audit and badge is going to give you the level of accessibility security that you desire (and are willing to pay for).

3. Does the badge bring value to the user using the website?

It’s more important for a website to be usable and an experience that users would want to repeat than to have a badge put on it.

In my experience, once the user has arrived at a website they will work out pretty quickly whether the site is accessible to their needs.

The presence of a badge on the site doesn’t really help. On the contrary, if a person with a disability can’t use your website, but sees a badge on it, that person is likely to become confused and annoyed – they know they aren’t getting a good experience, but some organisation out there that claims to speak on their behalf has said they should be.

What is useful is the testimonial aspect of the badge. To give an example: if I were a person with a disability in Hong Kong and I was looking for an online retailer, what would really help me would be the equivalent of a price comparison website that informed me of the accessibility badges that various competitor retailers had achieved. I’d then be able to see which was the most accessible, and use that information to help me decide which websites I would want to visit.

Accessibility badges are more like a Which? Product Review or Kitemark – the mark is of little value when you’ve already bought the product (or, in this case, visited the website); you need it when you are choosing which of the products that are available (websites to visit) will be most suitable for your particular set of needs and preferences.

So, when you’re choosing an organisation to audit and accredit your site, ask if they will add a link to your site to a directory of sites they have accredited and ask how well they promote that directory.

If the Hong Kong government wishes to “show appreciation to businesses and organizations for making their websites accessible” they should create an accessible directory of the sites they accredit, as this would be a useful guide for the disabled people the government wishes to help, and will help drive disabled people to the sites that have done the work to gain the ‘Gold’ level.

After all, ‘gaining more users’ is a much more tangible reward than ‘appreciation’.

Of course, the creation of one directory that amalgamated details of all accredited websites in each country, whichever accreditation badge they achieved, would be even more useful for users and organisations.

Such a directory wouldn’t be difficult to create, and – based on the limited number of organisations that have taken the trouble to get their websites accredited so far – wouldn’t currently be that difficult to maintain.

However, the value of such an amalgamated directory to users is constrained by one complicating factor – with so many different badges being used, will users be able to work out what each badge means?

4. What does the badge actually mean? – what were the testing metrics

The most important ‘meaning’ of a badge is the testing that it summarises. Its value is totally dependent on the value of the metrics being tested against.

Different badges differ in the checkpoints (usually known as ‘success criteria’) that sites are tested against:

• WCAG 2.0 A, AA and AAA badges are awarded for conformance with WCAG 2.0’s success criteria, which are allocated to different conformance levels.
• Many other badges are effectively critiques of the relative importance of the various success criteria that make up the different WCAG 2.0 levels, and WCAG’s insistence that sites achieve all success criteria at a given level. For example: the Hong Kong Scheme’s Gold badge includes a mixture of some WCAG 2.0 AA and some WCAG 2.0 AAA checkpoints. This is no doubt based – as are the metrics behind other badges – on the views and experience of those who set up the scheme on which checkpoints really matter to disabled people, and which may be too costly to implement in practice.

Secondly, badges differ in whether testing is done solely by experts, or includes some testing by disabled users:

• Some badges just test the technical accessibility of sites – such as general WCAG 2.0 audits and RNIB See It Right and Surf Right badges. However, as any accessibility specialist worth their salt, and recent academic research, can tell you, WCAG 2.0 audits can miss numerous problems that disabled people experience with web pages.
• Thankfully, an increasing number of badges – such as JISC TechDis Accessibility Passport, DAC’s Accreditation, Shaw Trust’s Accreditation and RNIB’s logo standard with UseAbility – also include user-testing by people with disabilities, to test whether or not good technical accessibility results in usable experiences for disabled and elderly people.

So, when you’re choosing an organisation to audit and accredit your site, ask what success criteria they will audit against, and whether they will include any user-testing with people with disabilities.

5. What does the badge actually mean? – transparency and ease of understanding

The second important aspect of the ‘meaning’ of a badge is whether people with disabilities can easily understand if the badge will predict whether a site will work for their particular needs.

While individuals may be interested in how well a website meets the needs of all people with disabilities, they are much more likely to be interested in whether it will meet their own particular needs.

And this is where almost all badges fall down.

Different groups of disabled people have very many different needs from each other. So, ideally, a useful badge would indicate clearly whether a site is suitable for a particular group’s needs.

Unfortunately, no badge that I know of does this, other than JISC TechDis’ accessibility passport for eLearning resources, which includes information on how accessible a resource is for people with different types of disability.

WCAG 2.0 conformance levels and all other accessibility badges mix up the success criteria that different disabled groups are particularly sensitive to. This means disabled people need to understand which success criteria are particularly important to their needs, and then work out what level each accreditation scheme assigns those success criteria to.

This means it is essential for accessibility accreditation badges to be transparent about what they mean.

DAC’s clickable accreditation badges (for example: see the DAC accredited badge at the bottom of the WY Metro Accessibility page) that link to a Certificate of Accreditation explaining what was tested, and when, is a good start.

Ideally such certificates would also include more details of the success criteria tested, with any deficiencies in the site’s accessibility – arranged by disability/impairment – clearly and simply highlighted, as recommended in BS 8878’s advice on How to Write an Effective Accessibility Statement.

So, when you’re choosing an organisation to audit and accredit your site, ask what mechanisms they will provide for disabled people to easily understand how the badge will predict whether a site will work for their particular needs.

Final thoughts – will the Hong Kong scheme impact accreditation more widely?

The Hong Kong government scheme is a bold and interesting initiative, and could be a great chance for its web industry to move forwards in its understanding and implementation of accessibility and inclusive design.

It will be useful in the future to see what the results of the scheme are:

• Whether free audit and accreditation results in more accessible sites in Hong Kong.
• Whether there are lessons that can be shared with other governments considering doing something similar.
• What impact (if any) it has on other organisations that provide accreditation services.

I’ll certainly be tracking what information they release later this year.