You'd need some algorithm that works on continuous variables, that is, #44E205 should rank closely to #44E106. Somehow you need to learn a function over rgb sampling only some of the points. Maybe you average over votes for colors that are close to a color in rgb space.

16 million collars aren't that many compared to large-scale internet systems. If you got 160 million+ votes you'd start getting meaningful results without any averaging.