It’s pretty hard to explain in short. But estimation algorithms don’t work on angles for example. For an algorithm an angle is just a number. So the difference between 359 and 1 degree looks pretty big. It is also not possible to just put a rotation matrix inside a state, because the algorithm has no concept of orthonormality. But if you define the change of a state in terms of vectors and the state itself as a rotation matrix you can just use off the shelf estimation algorithms like least squares and Kalman filters. You just need to redefine the plus and minus operators. Which is nothing you need a project for. You can just do it yourself.