The threat model isn't "what if someone tells everyone their username and password", it's "What if someone gets your username and password". I think most of the time people have one number they connect to, and everything else will be fringe cases. If you're worried they'll forget what phone number it's sent to, you can do the same thing you do with other verification/reset loops - "Enter your e-mail address and we'll send you an e-mail with the phone number you used". It might not always be the best authentication method, but at this point almost all authentication falls back to "I control the e-mail address that I controlled when I started the account" at this point anyway.

Plus, at the very least you could just reveal the last 2 digits of the phone number upon request. That's still side-channel data leakage, but at least it's much more contained.