This repository contains the official PyTorch implementation of the paper βTALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sightβ paper (NeurIPS 2024) by Hyun-Kurl Jang](https://blue-531.github.io/ ) , [Jihun Kim and Hyeokjun Kweon*.
(* denotes equal contribution.)
Our main idea is simple yet effective: an observation made at one moment could serve as supervision for the SSC prediction at another moment. While traveling through an environment, an autonomous vehicle can continuously observe the overall scene structures, including objects that were previously occluded (or will be occluded later), which are concrete guidances for the adaptation of scene completion. Given the characteristics of the LiDAR sensor, an observation of a point at a specific spatial location at a specific moment confirms not only the occupation at that location itself but also the absence of obstacles along the line of sight from the sensor to that location. The proposed method, named Test-time Adaptation via Line of Sight (TALoS) , is designed to explicitly leverage these characteristics, obtaining self-supervision for geometric completion. Additionally, we extend the TALoS framework for semantic recognition, another key goal of SSC, by collecting the reliable regions only among the semantic segmentation results predicted at each moment. Further, to leverage valuable future information that is not accessible at the time of the current update, we devise a novel dual optimization scheme involving the model gradually updating across the temporal dimension.
./
βββ
βββ ...
βββ model_load_dir
βββpretrained.pth
βββ dataset/
βββsequences
βββ 00/
β βββ velodyne/
| | βββ 000000.bin
| | βββ 000001.bin
| | βββ ...
β βββ labels/
| βββ 000000.label
| βββ 000001.label
| βββ ...
β βββ voxels/
| βββ 000000.bin
| βββ 000000.label
| βββ 000000.invalid
| βββ 000000.occluded
| βββ 000001.bin
| βββ 000001.label
| βββ 000001.invalid
| βββ 000001.occluded
| βββ ...
βββ 08/ # for validation
βββ 11/ # 11-21 for testing
βββ 21/
βββ ...
./model_load_dir
. [link]./experiments/baseline
for comparison. [link]python run_tta_val.py --do_adapt --do_cont --use_los --use_pgt
python run_tta_test.py --do_adapt --do_cont --use_los --use_pgt --sq_num={sequence number}
To evaluate test sequences in SemanticKITTI, you should submit the generated predictions to link. After generate predictions, prepare your submission in the designated format, as described in the competition page. Use the validation script from the semantic-kitti-api to ensure that the folder structure and number of label files in the zip file is correct.
We thanks for the open source project SCPNet.