How to remove scaffolding

[Dorch16]

I place scaffolding and I can remove it while I'm placing it with the e key, however if it is there for any length of time there is no way to remove it. I have scaffolding all over my property it looks horrible lol. Any help please - note: I have ran around the scaffolding over and over and there is no click e to remove scaffold message. bug? Great game by the way!

[TrompeL'Oeil]

Press F3, point to scaffolding, press e.