hell if I know!! Sorry. I've used the vegan package for some analyses, but I've mostly used Manhattan and cosine metrics. I just wanted to bring up the idea that there are a lot of metrics out there that may not be generally appreciated.
Claude Opus says "There are a few good distance metrics commonly used with latent embeddings to promote diversity and prevent model collapse:
1. Euclidean Distance (L2 Distance)
2. Cosine distance
3. Kullback-Leibler (KL) Divergence:
KL divergence quantifies how much one probability distribution differs from another. It can be used to measure the difference between the distributions of latent embeddings. Minimizing KL divergence as a diversity loss would encourage the embedding distribution to be more uniform.
4. Maximum Mean Discrepancy (MMD):
MMD measures the difference between two distributions by comparing their moments (mean, variance, etc.) in a reproducing kernel Hilbert space. It's useful for comparing high-dimensional distributions like those of embeddings. MMD loss promotes diversity by penalizing embeddings that are too clustered together.
5. Gaussian Annulus Loss:
This loss function encourages embeddings to lie within an annulus (ring) in the latent space defined by two Gaussian distributions. It promotes uniformity in the embedding norms while allowing angular diversity. This can be effective at preventing collapse to a single point.
But I haven't checked for hallucinations.
