...until you get presidential candidates campaigning on the promise that they'll fill SCOTUS seats with the candidates of their alignment. It's practically an election by proxy.
But we have that now. The idea of setting a term limit for Supreme Court justices is that it would minimize the effect of politicized appointments. Assuming the party that gets to make the appointments shifts over time, the makeup of the court is likely to be more balanced overall.