So the way out of this is that you have the constitutional right to confront your accuser in court. When accused by a piece of software that generally means they have to disclose the source code and explain how it came to its answers.

Not many people have exercised this right with respect to DUI breathalyzers but it exists and was affirmed by the Supreme Court. And it will also apply to AI.