Not quite math, but TAOCP was like that for me: rather than having to force myself to work through it, I actually looked forward to spending time reading it or working the exercises.
If you think you'd like TAOCP, but with just the math, not the programming (hah), take a look at Concrete Mathematics by Graham, Knuth, and Patashnik. I haven't read anything else by Graham or Patashnik, so I don't know how much their voice comes through, but Knuth's playfulness and joy in the material comes through strongly. It's a blast just to read for fun.
I must be built differently, I found Concrete Mathematics insanely inapproachable. I have a math stack exchange question where I ask for clarity on WTF he means on one question and got upvotes and comments from his students who were similarly flummoxed taking the course from him in real life!
