First, let me start by saying that this is a good article and sheds light on one of the challenges of accessibility adoption.
All of tools like lighthouse, axe-core etc. run a subset of tests that gives a false sense of security about accessibility. Similarly, tools like Accessibility Insights for Web has a fast-pass option, which does the same thing where it runs a subset of tests to catch the most common issues on a website.
But it does not and cannot(at this moment) catch all the issues that require semantic analysis of a website, like checking that the alt text on an image has meaningful text. For tests like those, a human is needed to perform a comprehensive assessment, something like Accessibility insights for web offers as an Assessment option.
In my opinion, all of the tools are doing one thing good and that is raise awareness about the problems that users with a disability face daily, when trying to use a website. They are making accessibility a must. Tools still needs more work and I feel confident that it will continue to improve. It all kind of comes down to how much time a development team puts in to make their website completely accessible, which ideally every team should budget and plan for.
Where I am, we use axe-core as part of our automated integration testing. Any change that regresses on accessibility (i.e. new severe a11y issue etc) is blocked from submission in the same way that it would be blocked from submission by causing tests to fail.
I find this useful as it prevents "accessibility rot" where people think "Oh we'll sort out the aria stuff later".
But yeah I agree that a human test is best. I think of axe et al as an equivalent of a "static analysis" that can pick out the obvious mistakes, but wont
understand the dynamic nature of the application - it is after all great for flagging the "easy" problems, allowing the human tester to focus on the major a11y issues without ending up raising hundreds of bugs for every button and link and colour combination that should have been sorted out way before any human testing is involved.
I usually rephrase the same point as: "Automatic testing is able to tell you if a page is inaccessible. However, it cannot tell you if a page is accessible."
I would rather not install Chrome at all on my windows machine. I use Brave (a chromium based browser) and with right extensions, I can see myself using the new Chromium based Edge.
I bet that will be the thought process for a lot of general people using browsers. Chrome will loose its market share to Edge for sure.
All of tools like lighthouse, axe-core etc. run a subset of tests that gives a false sense of security about accessibility. Similarly, tools like Accessibility Insights for Web has a fast-pass option, which does the same thing where it runs a subset of tests to catch the most common issues on a website.
But it does not and cannot(at this moment) catch all the issues that require semantic analysis of a website, like checking that the alt text on an image has meaningful text. For tests like those, a human is needed to perform a comprehensive assessment, something like Accessibility insights for web offers as an Assessment option.
In my opinion, all of the tools are doing one thing good and that is raise awareness about the problems that users with a disability face daily, when trying to use a website. They are making accessibility a must. Tools still needs more work and I feel confident that it will continue to improve. It all kind of comes down to how much time a development team puts in to make their website completely accessible, which ideally every team should budget and plan for.