Underwater Spatial Intelligence

USI-Brain

Enhancing Underwater Spatial Intelligence via
Spatial Data-Driven Vision-Language Modeling

Research Project Vision-Language Modeling
Zhiliang Zhu1,*,† Zhiqiang Zhao1,* Xie’an Jiang1,* Yingxin Lai2 Kangye Zheng1 Dawei Zhao1,‡
1 Harbin Engineering University 2 Xiamen University

* Equal contribution    † Project leader    ‡ Corresponding author

Explore the paper

Abstract

Underwater spatial reasoning depends on object boundaries and geometric cues that are obscured by scattering, color distortion, and low visibility. Existing underwater resources capture these cues through images, annotations, and depth maps, but provide fragmented supervision for vision-language models.

We present a spatial data engine that connects visual appearance with object-level geometry through enhancement, 2D and geometry-assisted question generation, and expert filtering. It produces 4.27M QA pairs across five capability groups. We use this corpus to train USI-Brain and introduce an Underwater Prior-guided Rectifier (UPR) that applies gated corrections to visual features.

USI-Bench evaluates the resulting capabilities with 6,525 held-out QA pairs. Across NautData, UWBench, and MarineEval, USI-Brain achieves the best or joint-best score on 28 of 32 metrics.

On the 15 USI-Bench tasks, USI-Brain improves over Qwen3-VL-8B-Instruct by 5.76 mean points, rising to 7.63 points with UPR. Controlled ablations separate the source of these gains: geometry-assisted supervision improves depth and ordering, HUE-derived supervision improves all 15 tasks, and UPR contributes most strongly to global scene description. Both model variants are evaluated directly on raw underwater images.

01

Appearance is unreliable

Wavelength-dependent attenuation shifts color, while backscatter, blur, and low illumination erase boundaries and texture.

02

Geometry is fragmented

Existing resources separate images, instance annotations, and depth, leaving object-linked spatial supervision incomplete.

03

Understanding must be joint

Real underwater tasks combine localization, 2D relations, depth, degradation awareness, and ecological context.

Overview of the underwater spatial data engine.
Figure 1. The underwater spatial data engine enhances heterogeneous inputs, generates 2D and geometry-assisted 3D QA, organizes them into five capability groups, and applies expert filtering. Select the figure to inspect it at full resolution.
4.27MSpatial QA pairs
6,525USI-Bench QA pairs
15Task-level metrics
28 / 32Best or joint-best

Three pieces, one spatial system

The work treats underwater spatial intelligence as a connected data, model, and evaluation problem rather than an isolated visual-language task.

01

Spatial Data Engine

Connects enhanced imagery, instance annotations, object-linked depth, and reconstructed scenes into 4.27M quality-controlled QA pairs spanning five capability groups.

02

USI-Brain + UPR

Adapts Qwen3-VL-8B-Instruct with structured supervision. UPR models scattering-related bias and absorption recovery, then applies a learned gated residual correction to visual tokens.

03

USI-Bench

Measures five connected capabilities through 6,525 held-out questions, 15 task metrics, source-level isolation, duplicate screening, and a complete audit by 15 annotators.

Method

The framework couples supervision construction with visual-feature adaptation. Geometry is used during data construction; inference only requires raw images and questions.

  1. 01Aggregate sources

    About 540K visual samples plus roughly 400 reconstructed Atlantis scenes.

  2. 02Enhance evidence

    HUE restores color, illumination, edges, and texture for supervision construction.

  3. 03Generate 2D QA

    Identity, attributes, locations, relations, visibility, and scene content.

  4. 04Lift into 3D

    Depth samples and reconstructed boxes support ordering, occlusion, distance, and viewpoint.

  5. 05Organize and review

    Five capability groups followed by expert selection, correction, and deduplication.

Supervision at scale

From fragmented resources to object-linked spatial evidence

The engine brings heterogeneous images, video frames, instance labels, depth, and reconstructed geometry into one QA representation while preserving the connection between an object, its image region, and its spatial cues.

Data foundation

A broad visual source pool

Approximately 540K visual samples are drawn from the ten datasets aggregated in NautData together with CoralVQA, WebUOT-1M, and about 400 reconstructed Atlantis scenes. The sources cover varied habitats, objects, visibility conditions, and annotation formats.

Candidate QA pairs are organized by capability and reviewed for answer evidence, localization consistency, and redundancy before entering the final corpus.

Model adaptation

One objective across 2D and 3D reasoning

USI-Brain is initialized from Qwen3-VL-8B-Instruct. The vision encoder, multimodal merger, and language model are jointly fine-tuned on the combined 2D and geometry-assisted corpus for two epochs.

The model uses the standard QA cross-entropy objective without auxiliary distillation, allowing the component studies to attribute gains directly to supervision coverage and UPR.

HUE

Restore visual evidence

Degradation maps for color shift, scattering, and blur condition two paths: one corrects color and illumination, while the other restores local edges and texture. A learned pixel-wise map selects their balance at every location.

GeoQA

Lift objects into geometry

Object masks or boxes collect aligned depth samples. Camera intrinsics back-project them into 3D, where fitted object boxes provide supervision for depth, ordering, occlusion, distance, viewpoint, and stratification.

UPR

Rectify visual features

A DINOv2 prior encoder estimates scattering-related bias and absorption-recovery gain. A learned gate controls the correction, then raw and rectified visual sequences are interleaved for the language model.

Architecture of USI-Brain and the Underwater Prior-guided Rectifier.
Figure 2. UPR estimates scattering-related bias and absorption-recovery gain, then rectifies visual tokens through gated residual fusion. In the fixed 512×512 setting, the 256-token raw sequence and 256-token corrected sequence are interleaved into 512 visual tokens.
Training configuration

Full-stack adaptation at scale

Training uses mixed precision, gradient checkpointing, SDPA, and FSDP full sharding. The language model and vision encoder use separate learning rates, while the model, merger, and visual stack remain trainable.

Training corpus
4.27M QA
Optimization
2 epochs
Compute
64 × B200
Full run
~25 hours

USI-Bench

A diagnostic benchmark for the spatial and semantic abilities required to understand complex underwater scenes.

B138.7%

Object Grounding
& Understanding

B217.5%

2D Relations
& Layout

B320.0%

3D Depth
& Stratification

B49.5%

Degradation
& Perception

B514.4%

Scene
& Ecology

Source-level isolation

Images, video frames, and all processed derivatives remain in the same split. Test sources from USI-Bench and all external benchmarks are excluded from training-QA construction.

Duplicate screening

Source IDs and QA links are checked alongside SHA-256 and perceptual hashes to reduce train-test leakage.

Full human audit

Fifteen independent annotators audited all 6,525 items. The initial 10% uncertain and 7% incorrect entries were revised before evaluation.

Task-aware scoring

Evaluation combines accuracy, Acc@IoU0.5, Recall@IoU0.5, Kendall's tau, macro-F1, and three independent semantic judges instead of reducing every capability to one generic metric.

Figure 3. Composition and representative question-answer examples across the five USI-Bench capability groups.

Results

USI-Brain improves spatial understanding on its internal diagnostic benchmark and generalizes across three external underwater evaluation suites.

USI-Brain+5.76

mean points over the Qwen3-VL-8B-Instruct backbone

Scene description+18.0

incremental points from UPR under matched training data

What the experiments show

Gains follow the supervision that was added

The main results and controlled removals provide complementary evidence: the corpus improves broad task coverage, geometry targets spatial ordering, and feature rectification most strongly affects global scene understanding.

Corpus transfer

12 of 15 tasks improve without UPR

USI-Brain gains +17.0 in captioning, +15.7 in structural perception, and +10.1 in ordering over the Qwen3-VL-8B-Instruct backbone.

Feature rectification

UPR changes the capability profile

USI-Brain + UPR improves 13 of 15 tasks and reaches the best or joint-best listed result on seven. Scene description rises from 47.3 to 65.3 under matched training data.

External transfer

The gains extend beyond USI-Bench

NautData VQA accuracy rises from 0.381 to 0.478, UWBench referring Acc@IoU0.7 improves by 13.87 points, and MarineEval spatial-relation assessment improves by 15 points.

Interactive result explorer

External benchmark coverage

Best or joint-best score within each paper comparison pool.

Switch benchmarks to update both the chart and the complete result table.

Reported values External benchmark coverage

Sole-best and joint-best counts from the paper comparison pools.

External benchmark coverage
Benchmark Sole best Joint best Total Coverage
NautData707100%
UWBench VQA60875%
UWBench Referring505100%
MarineEval911283.3%
Best or joint-best metric ratio across external benchmarks.
Figure 4. Percentage of reported metrics on which USI-Brain achieves the best or joint-best score in each comparison pool.
External benchmarks

28 / 32

best or joint-best metrics, with a mean signed advantage of +9.99 points over the strongest baseline for each metric.

NautData
7 / 7
UWBench VQA
6 / 8
UWBench Referring
5 / 5
MarineEval
10 / 12
Geometry-assisted QA+4.8 Depth · +6.1 Ordering

Object-linked geometry also adds 3.8 points in relation reasoning and 2.9 in distance estimation when UPR is disabled.

HUE-derived supervisionImproves all 15 tasks

Under matched GeoQA training, HUE adds 2.6 points in grounding, 3.3 in structural perception, and 2.2 in scene description—even though evaluation uses raw images.

Underwater Prior-guided Rectifier+18.0 Scene description

With the same 4.27M-QA corpus, UPR also adds 1.3 points in grounding, 1.1 in depth, and 0.9 in ordering.

Inference efficiency

A practical baseline and an explicit UPR trade-off

Single-image inference uses raw inputs and batch size 1 on an RTX 3090. UPR adds a prior encoder and doubles the post-merger visual sequence, improving scene understanding at a measurable latency and memory cost.

USI-Brain0.84 s

16.57 GiB peak memory
35.86 decode tokens/s

USI-Brain + UPR1.63 s

21.80 GiB peak memory
34.50 decode tokens/s

Citation

If you find this work useful, please consider citing the paper.

BibTeX
@article{zhu2026usibrain,
  title   = {USI-Brain: Enhancing Underwater Spatial Intelligence
             via Spatial Data-Driven Vision-Language Modeling},
  author  = {Zhu, Zhiliang and Zhao, Zhiqiang and Jiang, Xie'an and
             Lai, Yingxin and Zheng, Kangye and Zhao, Dawei},
  year    = {2026}
}