Well, sure, it's possible my consciousness is one that's travelled along every single branch where the backwards-time-travel didn't happen, but that strikes me as extraordinarily unlikely if there have been even only a fifty such attempts in all human (future) history.
What I mean is when a particle travels back in time, the universe branches forward in parallel, from that particle, at the instant it arrives.
This resolves all paradoxes as the independently instantiated time streams can't interact.
I believe there are present theories of time/space that rely on this kind of idea.
It also might mean any time traveller could never get back to the exact "when" they came from. Though if there was a way to traverse parallel time streams, there'd be no paradox as the moment they arrived "back" would also branch.
Ie, time is always a tree branching, and traveling back in time doesn't change that?