I've done this exercise and generally what you want is a combination of your suggestion (which is elaborated by 'polyphora[0]) and 'p1necone's suggestion [1]
Polyphora mentioned CIELAB as just one example, and it's a good example. I believe state of the art these days is Oklab[2], talked about here[3]. I'd like to pull out a comment from 'jiggawatts in that discussion:
> This is a tour de force of colour theory, and should be mandatory reading for anyone serious about computer colour!
I completely agree.
With regards to 'p1necone's suggestion, k-nearest neighbors is one simple and relatively easy way to separate the colors into bins. I've only done this on a single image, but with multiple images maybe you could also k-nn bin the resulting colors from each image and only return bins which have multiple members.
The most naive of interpretations (e.g. averaging each of the R/G/B values) is generally not what we want.
Would be super cool to add some different averaging schemes to this.