A similar phenomenon happens with various Western speakers in Japan: https://www.tofugu.com/japanese/aizuchi-japanese-listening/

It's a bit different as "aizuchi" is merely signaling that you're being listened to, rather than an actual switching of speakers. "Enthusiastic listening" is how I've heard it described, and it can be a distraction if you're not used to it.