> But there's nothing illegal nor discriminatory about companies having their workforce be in the office.

Basically, this is the disparate impact argument. You could make a similar argument about having the office only be accessible by climbing a flight of stairs.