I think you are wrong, observers in different galaxies will agree on the inertial reference frame in which the background radiation is isotropic. The relative velocity between different galaxies will show up as different dipole components in the measured background radiation, but if you compensate for that, you should get the same inertial reference frame no matter from which galaxy you measured.

I'll admit to not being a physicist and to being easily confused by the subject, so I'll just use a silly little metaphor to check whether I'm thinking straight, and try to avoid terminology I don't fully understand (which, since I don't have a particularly strong understanding of even special relativity, means not using relativity terms at all):

Think of the universe as a rubber sheet over a table, being stretched in all directions. My understanding is that cosmic background radiation is on the rubber, stretching and moving with it just like all the matter is. If background radiation were on the table instead, there would be exactly one spot on the rubber that saw no motion relative to it, while spots very far away from that spot would see a huge amount of motion. If you went out far enough, you'd be seeing hard radiation from half the sky.

Or, as another view, though I find this one harder to think about, since here the expansion of the universe seems to confuse rather than clarify things: The background radiation you see at any given point is simply the photons radiated inward from a sphere of a very large radius centered on that point. Every point has its own such sphere, and if an observer sees a dipole component in its background radiation, it means that that observer is in motion relative to the average motion of the matter that those photons originally came out of.