This looks exactly what I am trying to achieve with RSS but never quite got to the point where it's usable enough to stick to it long term.

How does someone learn about problems like these? Is this being taught at universities (Advanced abstract algebra) or where would you recommend learning about such things?

Quantum Information Science classes now exist at most universities. If you have average linear algebra and probability theory knowledge, it is relatively easy to jump into them (without physics background). The Scott Aaronson lecture notes are pretty great: https://www.scottaaronson.com/qclec.pdf

A standard undergraduate quantum computing course should suffice. Most of them would follow Nielsen and Chuang's book.