And they just go right back to it, just wait till they destory your metal barricaded door that is barricaded on both sides with metal.

Only way around this that I've found is to just set up your survivors in a separate area from your personal house, if you plan on barricading any doors. They'll just break down any doors in their way if they decide to path through it. If you really need to block off a door in their area, put walls around it or park a car in front of the door and they'll stop trying to path through that area. It's kind of a janky way to work around the pathfinding, but it works.