The server would have to downscale the received stream. That's at least one round of downscaling for each participant, possibly more if you want to send different quality versions of the same stream down to different participants.
And while your device is quite capable of rescaling your stream -- even a lowest-tier smartphone will do it with no strain on its GPU -- its much more of a strain on the server to rescale 100's/1000's of stream simultaneously.
Additionally: AFAIK you can't easily rescale an encoded stream: you need to decode it, rescale it and then re-encode it. For every single stream! That would be horribly computationally expensive.
