TopoRefine: Plug-and-Play Topology-Aware Contour Refinement for Building Segmentation

Building segmentation has recently benefited from foundation segmentation models such as SAM, which offer strong generalization and scalable deployment through simple prompts. However, SAM-style models still produce pixel-wise masks, whose fragmented regions and irregular boundaries often fail to preserve the geometric regularity of buildings, making direct contour vectorization unreliable. Existing attempts to improve building polygonization typically modify the segmentation decoder or introduce additional polygon prediction branches, increasing adaptation costs and weakening the portability of pretrained foundation models. In this paper, we ask whether accurate building polygonization can be achieved directly from frozen segmentation outputs, without modifying or retraining the underlying model. To this end, we propose TopoRefine, a plug-and-play topology-aware contour refinement framework that converts predicted contours into geometry-consistent building polygons. Unlike conventional contour refinement methods, which follow a fixed-topology paradigm and can only adjust a predefined set of contour points, TopoRefine performs topology-adaptive polygon evolution by jointly modeling vertex insertion, vertex deletion, and coordinate refinement. This allows the refined polygon to correct structural errors inherited from frozen segmentation outputs and progressively align with the underlying building geometry. Extensive experiments on three building segmentation benchmarks demonstrate that TopoRefine generalizes across different segmentation pipelines and consistently improves both segmentation accuracy and geometric consistency, achieving notable gains such as +6.17\% AP and +3.53\% PolySim on SAM 3.