That is a good one and in better times might have seen them recuse themselves? Justices have recused themselves in the past for apparent possible economic conflict of interest - don't know about political conflict of interest.

"shall hold their Offices during good Behaviour" is probably what's meant to address political conflict with a specific person: They nominally can't be removed by the person who nominated them.