GaussianSelector: Lightweight Human-Guided Object Selection in 3D Gaussian Splatting

Abstract

Selecting a complete 3D object from a reconstructed scene with minimal user effort is essential for practical scene editing and embodied interaction. Existing 3DGS-based methods either retrain the Gaussian representation to embed per-object labels, or build dense multi-view SAM observations, both requiring heavy computation and dense viewpoint coverage that is rarely available in practice.

We present GaussianSelector, a training-free framework for interactive 3D object selection from sparse views and sparse scribble guidance. Operating directly on native Gaussian primitives, we coarsen dense Gaussians into geometrically coherent superpoints and construct a continuity-weighted graph using appearance and spatial cues. Sparse user scribbles are lifted into 3D via visibility-aware transmittance coverage, and selection is solved as a global graph-cut energy minimization that propagates sparse evidence to a complete 3D object. This design naturally supports multi-round refinement, where users iteratively correct the selection from additional viewpoints to progressively improve the result.

Experiments demonstrate that GaussianSelector achieves competitive selection quality against state-of-the-art multi-view SAM-based methods, while requiring significantly fewer interaction views and substantially lower computational overhead.

Video

Method

Method overview. GaussianSelector converts a reconstructed 3DGS scene into a superpoint graph, lifts foreground and background scribbles onto visible Gaussians via alpha-transmittance coverage, and solves a binary graph cut whose labels are broadcast back to the Gaussian primitives.

Scene encoding, computed once. Raw spherical-harmonic coefficients are an unreliable appearance signature, since the same rendered look can be written many ways. We describe each primitive by its Canonical Axis Color instead, group Gaussians into superpoints, and connect those into a $k$-NN graph whose edges carry a continuity weight $w_{ij}$ combining spatial, appearance, and opacity distance. A label boundary crossing a high-$w_{ij}$ edge is expensive, so cuts land where the scene is genuinely discontinuous. None of this depends on the user, so it is cached once per scene.

Lifting scribbles into 3D. Projected centers are a poor test of membership: a Gaussian inside a scribble may contribute almost nothing to those pixels, while one outside may bleed through its alpha-composited footprint. We instead measure the visible coverage of each Gaussian by a scribble mask $M$, as the expected overlap under that Gaussian's own rendering distribution

$$ \rho_i \;=\; \frac{\sum_{\boldsymbol{p}}\alpha_{i\boldsymbol{p}}\,T_{i\boldsymbol{p}}\,M(\boldsymbol{p})} {\sum_{\boldsymbol{p}}\alpha_{i\boldsymbol{p}}\,T_{i\boldsymbol{p}}+\epsilon}, $$

where $T_{i\boldsymbol{p}}$ is the accumulated transmittance before $G_i$. In words: if a pixel were sampled from wherever $G_i$ actually renders, how likely is it to land inside the scribble? Confident Gaussians become seeds and vote up to their superpoint.

Selection as a graph cut. Fitting one GMM per seed set gives every superpoint a radiance margin, the log-likelihood ratio between foreground and background. With the continuity prior, selection becomes a binary energy

$$ E(L) \;=\; \sum_{k\in\mathcal{V}_s} D_k(L_k) \;+\; \lambda \sum_{(i,j)\in\mathcal{E}_s} w_{ij}\,\mathbf{1}[L_i \neq L_j], $$

whose unary term $D_k$ turns that margin into a per-superpoint cost, and whose pairwise term charges $w_{ij}$ for every edge the boundary crosses. It is a submodular Potts energy, so an s-t min-cut recovers the exact global minimum. Each new round re-runs only the lifting, the fit, and the cut — never the scene encoding. Full derivations are in the paper.

Qualitative Comparison

Qualitative comparison on NVOS under various input view settings (NVOS scribbles, 1/2 additional views or interaction rounds).

Qualitative comparison on the 3D-OVS benchmark.

Qualitative comparison on 3D-OVS. Our method uses 5 interaction rounds; all baselines use all views.

Human-in-the-Loop Refinement

User interaction demonstration for human-in-the-loop object selection workflow. Each interaction round re-executes only visibility-aware scribble lifting, seed-conditioned likelihood estimation, and graph-cut inference. Users inspect the current selection, add a complementary scribble from an informative view, and see the result update within seconds.

Quantitative Results

Quantitative comparison on the NVOS benchmark. The best results are highlighted in bold, and the second-best results are underlined. Runtime is reported in minutes.

Average runtime comparison on all NVOS object selection tasks (in seconds). Preparation denotes scene-level preprocessing or optimization; inference refers to the query stage; training denotes feature field learning when required.

User Study

User study results (mean ± SD, best in bold). Across 12 participants on two object selection tasks, our scribble-based interaction gives the most precise user control, while all three methods are rated comparably on blind visual satisfaction. The decisive difference is response time: fast feedback lets users iteratively refine their input, yielding markedly better waiting acceptability and overall usability.

BibTeX

@misc{yang2026gaussianselectorlightweighthumanguidedobject,
      title={GaussianSelector: Lightweight Human-Guided Object Selection in 3D Gaussian Splatting with Graph Optimization}, 
      author={Baihan Yang and Tiexin Li and Yuheng Liu and Xin Lin and Xinke Li and Xiaohui Xie and Truong Nguyen},
      year={2026},
      eprint={2608.01492},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.01492}, 
}

GaussianSelector: Lightweight Human-Guided Object Selection in 3D Gaussian Splatting with Graph Optimization

Overview of GaussianSelector. We propose a plug-and-play interactive framework for object selection in 3DGS. Given user scribbles on a single view or a few views, GaussianSelector extracts 3D objects efficiently using graph-based optimization in the 3DGS space.