> humans decided they were "prime", i.e. most important, based on arbitrary considerations
No, not arbitrary considerations.
The term goes back at least to Euclid who investigated factorization of integers in his "Elements". He used the greek word "protos" that was later translated to Latin as "primus". It doesn't mean "most important", rather "first".
The idea is that primes are the "multiplicative building blocks" for other numbers, the "origin" or "first principles", because every integer factors into primes. When a mathematical object can be decomposed in some way, it is very natural to study the irreducible blocks, because many questions boil down to them.
The obsession with prime numbers (humans decided they were "prime", i.e. most important, based on arbitrary considerations).
It seems like a version of astrology to me.
Am I wrong? I'd be happy to be proved wrong.