Some ideas are:
- You branch off into another timeline and your actions make no difference to the previous timeline
- You’ve already taken said actions but just didn’t know about it so nothing changes
- Actions taken can have an effect (so you could suddenly erase yourself if you killed your parents)
- Only “nexus” or fixed events really matter, the timeline will sort itself out for minor changes
- something else entirely
I can think of a couple ways around that, the easiest is that I actually think time travel is impossible. (Like this for example)
If it’s not impossible, then single-timeline travel probably is, and all (backwards) travel would start a new timeline.
Short of that, maybe something ridiculous would have happened when the traveler “first” went back, like one of them tripping or whatever, and the handshake they agreed to try didn’t go as planned, and then “still” didn’t the traveler’s second time. Basically this.