My reading is the server with zsync can be any http server, client is zsync. Rsync on the other hand needs to use rsync on the server, and it scales badly because it does most work on the server.

That's exactly it. Zsync works only once per file, creates a signature in a second file and that's it. They are both distributed with an HTTP server, the most scalable way of centrally distributing data. The client fetches the signature, checks what part is missing and downloads it with Range: headers.