Appearance is unreliable
Wavelength-dependent attenuation shifts color, while backscatter, blur, and low illumination erase boundaries and texture.
Underwater spatial reasoning depends on object boundaries and geometric cues that are obscured by scattering, color distortion, and low visibility. Existing underwater resources capture these cues through images, annotations, and depth maps, but provide fragmented supervision for vision-language models.
We present a spatial data engine that connects visual appearance with object-level geometry through enhancement, 2D and geometry-assisted question generation, and expert filtering. It produces 4.27M QA pairs across five capability groups. We use this corpus to train USI-Brain and introduce an Underwater Prior-guided Rectifier (UPR) that applies gated corrections to visual features.
USI-Bench evaluates the resulting capabilities with 6,525 held-out QA pairs. Across NautData, UWBench, and MarineEval, USI-Brain achieves the best or joint-best score on 28 of 32 metrics.
On the 15 USI-Bench tasks, USI-Brain improves over Qwen3-VL-8B-Instruct by 5.76 mean points, rising to 7.63 points with UPR. Controlled ablations separate the source of these gains: geometry-assisted supervision improves depth and ordering, HUE-derived supervision improves all 15 tasks, and UPR contributes most strongly to global scene description. Both model variants are evaluated directly on raw underwater images.
Wavelength-dependent attenuation shifts color, while backscatter, blur, and low illumination erase boundaries and texture.
Existing resources separate images, instance annotations, and depth, leaving object-linked spatial supervision incomplete.
Real underwater tasks combine localization, 2D relations, depth, degradation awareness, and ecological context.
The work treats underwater spatial intelligence as a connected data, model, and evaluation problem rather than an isolated visual-language task.
Connects enhanced imagery, instance annotations, object-linked depth, and reconstructed scenes into 4.27M quality-controlled QA pairs spanning five capability groups.
Adapts Qwen3-VL-8B-Instruct with structured supervision. UPR models scattering-related bias and absorption recovery, then applies a learned gated residual correction to visual tokens.
Measures five connected capabilities through 6,525 held-out questions, 15 task metrics, source-level isolation, duplicate screening, and a complete audit by 15 annotators.
The framework couples supervision construction with visual-feature adaptation. Geometry is used during data construction; inference only requires raw images and questions.
About 540K visual samples plus roughly 400 reconstructed Atlantis scenes.
HUE restores color, illumination, edges, and texture for supervision construction.
Identity, attributes, locations, relations, visibility, and scene content.
Depth samples and reconstructed boxes support ordering, occlusion, distance, and viewpoint.
Five capability groups followed by expert selection, correction, and deduplication.
The engine brings heterogeneous images, video frames, instance labels, depth, and reconstructed geometry into one QA representation while preserving the connection between an object, its image region, and its spatial cues.
Approximately 540K visual samples are drawn from the ten datasets aggregated in NautData together with CoralVQA, WebUOT-1M, and about 400 reconstructed Atlantis scenes. The sources cover varied habitats, objects, visibility conditions, and annotation formats.
Candidate QA pairs are organized by capability and reviewed for answer evidence, localization consistency, and redundancy before entering the final corpus.
USI-Brain is initialized from Qwen3-VL-8B-Instruct. The vision encoder, multimodal merger, and language model are jointly fine-tuned on the combined 2D and geometry-assisted corpus for two epochs.
The model uses the standard QA cross-entropy objective without auxiliary distillation, allowing the component studies to attribute gains directly to supervision coverage and UPR.
Degradation maps for color shift, scattering, and blur condition two paths: one corrects color and illumination, while the other restores local edges and texture. A learned pixel-wise map selects their balance at every location.
Object masks or boxes collect aligned depth samples. Camera intrinsics back-project them into 3D, where fitted object boxes provide supervision for depth, ordering, occlusion, distance, viewpoint, and stratification.
A DINOv2 prior encoder estimates scattering-related bias and absorption-recovery gain. A learned gate controls the correction, then raw and rectified visual sequences are interleaved for the language model.
Training uses mixed precision, gradient checkpointing, SDPA, and FSDP full sharding. The language model and vision encoder use separate learning rates, while the model, merger, and visual stack remain trainable.
A diagnostic benchmark for the spatial and semantic abilities required to understand complex underwater scenes.
Object Grounding
& Understanding
2D Relations
& Layout
3D Depth
& Stratification
Degradation
& Perception
Scene
& Ecology
Images, video frames, and all processed derivatives remain in the same split. Test sources from USI-Bench and all external benchmarks are excluded from training-QA construction.
Source IDs and QA links are checked alongside SHA-256 and perceptual hashes to reduce train-test leakage.
Fifteen independent annotators audited all 6,525 items. The initial 10% uncertain and 7% incorrect entries were revised before evaluation.
Evaluation combines accuracy, Acc@IoU0.5, Recall@IoU0.5, Kendall's tau, macro-F1, and three independent semantic judges instead of reducing every capability to one generic metric.
Figure 3. Composition and representative question-answer examples across the five USI-Bench capability groups.
USI-Brain improves spatial understanding on its internal diagnostic benchmark and generalizes across three external underwater evaluation suites.
mean points over the Qwen3-VL-8B-Instruct backbone
mean points across all 15 USI-Bench tasks
incremental points from UPR under matched training data
The main results and controlled removals provide complementary evidence: the corpus improves broad task coverage, geometry targets spatial ordering, and feature rectification most strongly affects global scene understanding.
USI-Brain gains +17.0 in captioning, +15.7 in structural perception, and +10.1 in ordering over the Qwen3-VL-8B-Instruct backbone.
USI-Brain + UPR improves 13 of 15 tasks and reaches the best or joint-best listed result on seven. Scene description rises from 47.3 to 65.3 under matched training data.
NautData VQA accuracy rises from 0.381 to 0.478, UWBench referring Acc@IoU0.7 improves by 13.87 points, and MarineEval spatial-relation assessment improves by 15 points.
Best or joint-best score within each paper comparison pool.
Switch benchmarks to update both the chart and the complete result table.
The chart could not be rendered. The complete values remain available in the table below.
Sole-best and joint-best counts from the paper comparison pools.
| Benchmark | Sole best | Joint best | Total | Coverage |
|---|---|---|---|---|
| NautData | 7 | 0 | 7 | 100% |
| UWBench VQA | 6 | 0 | 8 | 75% |
| UWBench Referring | 5 | 0 | 5 | 100% |
| MarineEval | 9 | 1 | 12 | 83.3% |
best or joint-best metrics, with a mean signed advantage of +9.99 points over the strongest baseline for each metric.
Object-linked geometry also adds 3.8 points in relation reasoning and 2.9 in distance estimation when UPR is disabled.
Under matched GeoQA training, HUE adds 2.6 points in grounding, 3.3 in structural perception, and 2.2 in scene description—even though evaluation uses raw images.
With the same 4.27M-QA corpus, UPR also adds 1.3 points in grounding, 1.1 in depth, and 0.9 in ordering.
Single-image inference uses raw inputs and batch size 1 on an RTX 3090. UPR adds a prior encoder and doubles the post-merger visual sequence, improving scene understanding at a measurable latency and memory cost.
16.57 GiB peak memory
35.86 decode tokens/s
21.80 GiB peak memory
34.50 decode tokens/s
If you find this work useful, please consider citing the paper.
@article{zhu2026usibrain,
title = {USI-Brain: Enhancing Underwater Spatial Intelligence
via Spatial Data-Driven Vision-Language Modeling},
author = {Zhu, Zhiliang and Zhao, Zhiqiang and Jiang, Xie'an and
Lai, Yingxin and Zheng, Kangye and Zhao, Dawei},
year = {2026}
}