Pointers to a vec could also be invalidated at any point when adding a new element and causing a reallocation

(I think Miri will shout at you if you use a invalidated pointer here)