How far out can I build my walls?

[Ikit Claw]

The villagers houses are so spread out that I gotta wall off a huge space, I feel like enemies will spawn inside.
Wish we could move their houses so I can make it more compact.

[Posh]

from my experience they spawn really far away like almost out of render distance should be fine.