You are not taking it to the extreme. You are just discovering that "selfishness" is not hard-defined. It is relative, and thus one cannot measure it through different people since each one of them will perceive it differently.