Some people interpret the leaving and the coming back as distinct events. When you leave, they think you're giving up, either on resolving the situation or repairing the relationship at all. Until you come back, there's an anxiety-inducing chance that you won't.