While uncertainties are useful, they are certainly not enough. Eg: I can construct adversarial examples which are predicted incorrectly with high confidence.
The underlying reason why high confidence is not enough is that even strong/confident correlations could be misleading when seen in causal light — a black box model trained to predict credit performance might be very confident in rejecting loans for applicants from “poorer” zip codes and approving those from “richer” zip codes — even though those are not actual causes... therefore somebody could exploit the system by renting an address in a rich neighborhood for a couple of months when taking out a big loan (the analogue of adversarial examples).
I stated that rather than explainability, for most use cases, people just want calibrated and robust uncertainty estimates NOT low quality and uncalibrated uncertainties estimates.
Your example points to models that provide low quality uncertainty estimates, but that's not true for all deep learning models.
I believe it's these low quality uncertainty estimates that lead people to look toward "explainability" as a solution, but for the majority of use cases, I think people just want better uncertainty estimates so that they can "know when they're model doesn't know".
There are techniques now to get higher quality, calibrated, uncertainty estimates that don't suffer from the problems you mentioned and I've outlined these solutions in my posted link above.
Additionally if you're interested, there is some nice recent research from google on the subject:
Thanks for the links here and above; will take a look.
> calibrated and robust uncertainty estimates NOT low quality and uncalibrated uncertainties estimates.
Could you explain what you mean by “calibrated” and briefly summarize the essential idea behind what allows the learning of robust uncertainty estimates, if not a causal understanding?
If you haven’t already, look up work by Scholkopf, Janzing , Peters and co (over the last decade) for a justification of why causal reasoning is exactly what you want if you want to generalize across covariant earth/dataset shift (which is basically what the Google blog post is about).
Sure, but at the same time, there's the concept of adversarial examples not being the model's fit to noise, but simply being a different, human-imperceivable feature instead.
Considering that ML has access to orders-of-magnitude greater quantities and resolution of information that we do, why do we expect their decision structure to be intensely linked to our own decision process.
The underlying reason why high confidence is not enough is that even strong/confident correlations could be misleading when seen in causal light — a black box model trained to predict credit performance might be very confident in rejecting loans for applicants from “poorer” zip codes and approving those from “richer” zip codes — even though those are not actual causes... therefore somebody could exploit the system by renting an address in a rich neighborhood for a couple of months when taking out a big loan (the analogue of adversarial examples).