So after some time my workers started acting weird, they kept on going into self preservation and protecting allies. Which isn't possible, because there were no enemies inside the town. So when I looked where they kept on walking towards, it was the edge towards the valley. And the valley had quite a few bandits in it. Look at the red circles. Its where AI are located. The first is a caravan trader unable to attack his enemies. The second are the bandits locked inside the valley. Thirdly, there are bandits above the caravan trader, but they too are unable to get anywhere. If a valley is inaccessible, how can AI suddenly be in it? Worst of all, my workers shouldn't be trying to attack them if they are unreachable. Lastly for the first part, if AI does fall from an edge into a valley - they should be dead.
So thats part 1 of the bug. Part 2 is that AI uses slopes that are too steep to even walk on or walk onto. What happens when they get stuck or walk somewhere that should be inaccessible is one of the following:
1) They get stuck and are unable to move anywhere. (a known issue)
2) They fall down into areas that should be unreachable.
3) They disappear. See image: Instead the slopes should be unpassable and block AI from moving.
Ofcourse walking away from my town with all the workers and being away for a while makes the bandits go away, but it doesn't help if you want your town to produce 24/7 (figure of speech).