A sophisticated attacker might know that you were going to connect to a self-signed site, though. Interestingly though, private DNS (DoH, etc.) might help further shroud this fact from the attacker.

All in all, I'd say that the browser should still throw up a full-page warning because of the implications of TOFU, but it can be one where the "continue to site" option is clearly shown even to a naïve user, and not hidden behind a spoiler.