Usually they walk up to my outpost and stop a short distance outside my base or at my gates. It only happens if I keep the gates of my outpost closed. Traders/Raiders stop for days until I open the gates. Then they walk to some spot in my base or through my whole base to another gate where the event takes place.
The problem does not occur if I keep my gates open. But in my case that's not very practical - the base is located in skinner's roam where tons of hungry bandits swarm the base and the gate guards can't stop them. I decided to report this as five different Raids/Traders waited patiently at my outpost in my last game.