The guards patrols break constantly in the ghost camp

[Vahnkiljoy]

I have to keep saving and loading to even have a chance to ATTEMPT not killing anyone what the hell.

[DSideDaemons]

I'd suggest looking up this quest on youtube, if there is such a video, to see if you can do something different.

[Stu -1]

you're right, in my experience the guards randomly get stuck between being static, or wandering off on 'patrol'. However, when you enter the area, if the guard nearest you (facing the way you want to travel) and the one on the pathway to the left facing the building, move off on patrol. you should be able to just hug the left wall and make it to the small cliff edge you need to climb. i never had to reload this area more than once or twice on all my playthroughs, hopefully your luck picks up.

you could also try making a haze/greater haze potion if you have a save prior to entering the camp. another thing to note is if you want to properly complete this quest 100% successfully you have to head out of the camp and into the woods before you do the small stealth area. the first time i played it i missed this bit and had to reload, because other than some companion dialogue, there is no direction telling you to do this.

UPDATE

so i just went and watched this part on your stream. the third time, when both guards move and you ask if they were glitched. that was the ideal time to go. but you should have hugged the left wall instead of going to the right. i know, if you're gona include stealth then you shouldnt cripple it to having only one cheesy path to get to the goal. but stealth isn't a great mechanic in this game. but i think this is the issue you were having.