Yes. At least according to California law, you're always to allow faster traffic to move to left of you. Relatedly, it's highly recommended in the driving handbook to never pass on the right. Take it as a safety precaution or whatever you need, but I'm unaware of a good reason to drive in the HOV lane unless you're moving noticeably faster than traffic in the right lanes.
