Not sure about the US, but in the UK, if a manager agreed that you could remote work, and then at some point later, withn no justification (e.g. poor performance), and the company tried to back out of that, it would be construed as constructive dismissal, which can force a pay out.