Adding to Manor House Building Issue

[ⒺⒹⒾⓃ]

I decided to add walls to my manor house and retinue tower which I already built earlier....BUT the builders DESTROYED the manor house and retinue tower and decided to build it ALL over again, costing me ALL the materials AGAIN even for the buildings already there!