For the phone only route, it seems like you could still mostly automate it by going oldschool. Give the host an option to play the meeting code as a DTMF signal (or whatever) while the other person holds their phone near the mic.
Maybe I'm misunderstanding the use case then? I'm imagining something like:
A and B are on a phone call. A starts a video meeting. B goes to shortlink.dtmf or opens the app, which starts listening. A clicks "transmit room code" which goes over the existing phone connection. B's client hears the signal, decodes it, and gives them a link to join.