NexusDwin/sailswarm-lraspp-thermal

SailSwarm LRASPP thermal student (water/sky/obstacle segmentation on a FLIR Lepton 3, 160×120)

lraspp_mobilenet_v3_large (torchvision, ~3.2 M params) trained to segment obstacle / water / sky in long-wave infrared frames from a FLIR Lepton 3.0 (PureThermal, 160×120, ~57° HFOV) mounted on the SailSwarm autonomous-sailboat obstacle-detection box (University of Konstanz, Lake Constance). Companion to the RGB model NexusDwin/sailswarm-lraspp-student; same architecture, same 3-class convention {0 obstacle, 1 water, 2 sky}.

The point of this model: thermal is the night-time primary sensor for the boat, and no public maritime LWIR segmentation ground truth exists at this resolution. The labels are cross-modal: the RGB student's masks on the co-mounted fisheye camera, warped into the thermal frame (bearing-for-bearing under the measured linear thermal bearing model + a horizon-fit vertical offset; the two lenses share a rigid mount with a ~6 cm baseline, so parallax is <1° beyond 2.5 m).

What is in this repo

Folder	Generation	Data	Pinned holdout (under-pier clip 2026-07-08 16-44-56, n=752)
v1_day_2026-08-22/	JOINT recipe, day-only thermal	33,467 gray-Konstanz fisheye frames (six missions) + gray LaRS + ~10k warped real thermal frames, 60 epochs	acc 0.9090 / 0.9069 / 0.9080 (seeds 0/1/2), obstacle IoU 0.89, water IoU 0.46, sky 0.74
v2_night_2026-08-28/	same JOINT recipe, first dusk/night thermal (2026-08-26 outing: pontoon → open water → harbour dusk → night)	68,350 gray-Konstanz native frames (whole 2026 corpus) + gray LaRS + 114,501 total training pairs incl. ~50k warped thermal frames from 63 clips, 60 epochs	acc 0.9136 / 0.9139 / 0.9131 (seeds 0/1/2 → 0.9135 ± 0.0003), obstacle IoU 0.90, water IoU 0.54–0.55, sky 0.69

Per seed: thermal_joint_s<k>_best.pth (torchvision state dict) and thermal_joint_s<k>_160x120.onnx (fp32, opset 17, fixed 160×120, input image NCHW, output logits 1×3×120×160). train_all_seeds.log is the full training log. Seed variance ≤ ±0.001 in both generations — pick any; seed 0 is the one we evaluate against. Use v2_night_2026-08-28/ unless you specifically need the day-only model: it is +0.005 acc on the pinned holdout and is the only generation that has seen thermal frames captured after civil dusk. Water IoU +0.10 / sky −0.06 vs v1 on that holdout — the night frames shift where the model draws the water boundary under the pier; treat the sky drop as a known property of this eval scene (almost no sky visible), not a regression measurement. warp_fit_report.json is the per-clip fisheye→thermal warp fit used to make v2's labels.

Why "JOINT" — the lesson this model encodes

Trained on thermal alone (~1k frames of one dock), this architecture memorised scene layout (holdout 0.38–0.58 across seeds). Grayscale RGB from the same lake, trained jointly in the same batches as the thermal frames, is a stronger structural prior than more thermal data: 10k gray frames of one dock → 0.61 ± 0.02; 33k gray frames across six missions → 0.908 ± 0.001. Sequential fine-tuning from a gray-pretrained init does not reach this; joint batches do. Augmentations: random polarity inversion (p 0.5 — the Lepton's AGC flips warm/cold contrast between day and night) and ±12 px vertical shift (breaks horizon-row memorisation).

Preprocessing (must match)

Input is the upright, colourised 8-bit Lepton stream collapsed to luminance and replicated to 3 channels, then ImageNet-normalised — exactly what the --gray training flag did:

import onnxruntime as ort, numpy as np, cv2
sess = ort.InferenceSession("v1_day_2026-08-22/thermal_joint_s0_160x120.onnx",
                            providers=["CPUExecutionProvider"])
frame = cv2.imread("thermal.png")                       # 160x120 BGR from the PureThermal stream
g = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY).astype(np.float32) / 255.0
x = np.stack([g, g, g], 0)                              # gray x3, CHW
x = (x - np.array([0.485, 0.456, 0.406])[:, None, None]) / np.array([0.229, 0.224, 0.225])[:, None, None]
cls = sess.run(None, {"image": x[None].astype(np.float32)})[0][0].argmax(0)   # 120x160 class map

Our capture rotates the thermal frame 180° at the box (the module is mounted inverted); feed the model the upright frame. Repair dead sensor rows before inference if your Lepton has them (ours has row 21 stuck hot).

Limits — read before using

• v1 is day-only; v2 has one dusk/night outing (2026-08-26, ~4 h of Lepton footage, pseudo-labelled from the RGB model while the RGB model could still see). Frames after full dark are labelled from an RGB teacher operating near its own luminance floor — those labels are the weakest in the set. The pinned holdout is a daytime clip, so the numbers above do not measure night performance; audited night thermal ground truth does not exist yet.
• Water IoU 0.46 on the holdout is a property of the holdout scene (an under-pier clip with almost no visible water), not a measurement of open-water quality; there is no independent thermal ground truth yet — every label is a warped RGB pseudo-label, so systematic RGB-model errors (e.g. reflections read as obstacle) are inherited.
• fp32 only. Static int8 quantisation fails our pixel-agreement gate on this model (per-frame agreement min 40% on seed 0, worse on seeds 1–2) — recipe-independent fragility of weak thermal logits under quantisation. At 160×120 the fp32 ONNX is fast enough on a Raspberry Pi 4 CPU.
• Not validated for closed-loop navigation.

Provenance / license

Backbone: torchvision lraspp_mobilenet_v3_large (BSD-3, ImageNet-pretrained). Training data: LaRS (Žust et al., ICCV 2023; research/non-commercial — derived weights inherit its terms) as grayscale, plus Lake-Constance fisheye and Lepton footage captured by the SailSwarm project (University of Konstanz, Cluster of Excellence in Collective Behaviour) with pseudo-labels from NexusDwin/sailswarm-lraspp-student. Trained with scripts/gpu_seg/train_student.py (--gray --aug-polarity 0.5 --aug-vshift 12 --size 160x120) from the SailSwarm obstacle-detection repository.

v3_night_2026-09-02 (added 2026-09-09)

JOINT retrain of the thermal LRASPP student with the 2026-08-26 dusk/night outing in the thermal set (three seeds, 60 epochs, same recipe as v2). Pinned under-pier holdout (clip 2026-07-08 16-44-56, n=752): pixel accuracy 0.9148 / 0.9149 / 0.9146 = 0.9148 ± 0.0001, obstacle IoU 0.902 (v2_night: 0.9135 ± 0.0003; v1_day: 0.908). Seed 1 (thermal_night_s1_160x120.onnx, md5 d1add978…) is the copy deployed on the box as ~/ewasr/thermal_student_night_v3_s1_160x120.fp32.onnx: 25 fps at 160×120 on a Raspberry Pi 4 (2 threads, fp32; int8 fails the agreement gate for every thermal generation, so fp32 is the deployment format). Not yet a voter in the live fusion scorer; the shadow runtime consumes the fisheye student only (2026-09-09). Training log: night_ladder.log.

v4b_night_gated_2026-09-12 and v5_massmind_2026-09-12 (added 2026-09-15)

Both are warm-started from v3 (three seeds each, 60 epochs, same recipe) after the 2026-09-08 lake session, whose recording runs from daylight into true night. Every thermal pseudo-label is now gated on fisheye luminance (mean luminance >= 25 per chunk): after 20:15 the RGB teacher's frame is black and its mask reads 99.6 to 100 % obstacle, so an ungated retrain (v4, discarded) merely learned to paint everything obstacle. No night number can exist from cross-modal labels; the holdout below is the four darkest chunks that still have a valid teacher (19:56 to 20:11, dusk).

model	19:56	20:01	20:06	20:11	mean pixel acc
v3_night (never saw 2026-09-08)	0.858	0.883	0.876	0.719	0.834
v4b_night_gated	0.869	0.905	0.900	0.728	0.851
v5_massmind					0.844 (within seed noise of v4b)

• v4b: 31 gated 2026-09-08 thermal chunks + 31 gray-Konstanz + LaRS-gray. Modest, consistent dusk gain (water IoU 0.80 to 0.85 -> 0.90 to 0.95). Logs thermal_train_0908_s*.log.
• v5: the v4b mix plus the MassMIND LWIR training split (Boston-harbour USV, 7 pixel classes mapped to sky/water/obstacle). Konstanz dusk 0.844 (no change) but MassMIND test pixel acc 0.973 / obstacle IoU 0.85 (v3: 0.926 / 0.63): a broader maritime-LWIR model at no cost on our own domain. MassMIND is CC BY-NC-SA; derived weights inherit its terms.
• fp32 only (int8 fails the agreement gate on every thermal generation). None of these is deployed on the vessel; the box still runs v3 seed 1.
• The 20:11 chunk scores obstacle IoU about 0.04 for every model: the teacher there is already unreliable, which is the reason for the gate.

Record: docs/history/2026-09-12_night_retrain_aime.md sections 3 and 9 in the repository.