Aluode/Neuro_Splat

Neuro Splat — an artificial visual cortex (Gabor wave-packet VAE)

PerceptionLab / Antti Luode, with Claude (Opus 4.8), in dialogue with Gemini. Helsinki, June 2026.

Do not hype. Do not lie. Just show.

An image is not a million pixels. It is a smooth map and a scatter of small bright wave-packets dropped where the detail lives.

---

What this model is

This repo holds the weights (model.pt) for a VAE whose decoder is a differentiable Gabor wave-packet splatter. It is not a pixel-generating image model; it is a biologically-motivated representation experiment — an architecture that reproduces the shape of how V1 is thought to code images (localized Gabor atoms), used to test predictive coding and phase locking.

• Encoder (CNN): image → a latent "concept".
• Decoder (MLP): latent → the parameters of 512 localized Gabor atoms — position, scale, orientation, frequency, and a complex (a, b) coefficient per colour channel.
• Renderer: splats those wave-packets onto the canvas; the image is their sum.

Trained on Oxford Flowers-102 at 128×128 with 512 packets.

The blur is the prior; reality is the sharpness

Asked to generate a flower from a random latent, this model returns a soft, watercolour blob — and that is the correct, expected behaviour, not a failure. A generative prior with no input has to average over everything it cannot disambiguate, so it returns the low-frequency gist.

To see it work, you give it eyes. Hook it to a webcam and the blurry top-down prior collides with raw bottom-up reality; the live frame supplies the phase the prior could not guess, the packets lock to it, and the gist sharpens. That collision — top-down prediction meets bottom-up residual — is the predictive-coding loop, made live.

How to load it

You need the architecture from the ArtificialCortex repo (splat_generator.py defines SplatVAE).

import torch
from huggingface_hub import hf_hub_download
from splat_generator import SplatVAE   # from the ArtificialCortex repo (the_splat/)

ckpt = hf_hub_download("Aluode/Neuro_Splat", "model.pt")
model = SplatVAE(image_size=128, latent=128, num_packets=512, chunk=64)
model.load_state_dict(torch.load(ckpt, map_location="cpu"))
model.eval()

How to run it

pip install torch torchvision numpy opencv-python huggingface_hub

# perception — watch the packets phase-lock to a live webcam (panel 3)
python live_cortex_perception.py --model_path model.pt --image_size 128 --num_packets 512

# generation — the blurry priors from random latents (expected to be soft)
python splat_generator.py --mode sample --resume model.pt --image_size 128 --num_packets 512

--image_size 128 --num_packets 512 must match these weights, or the state_dict will not load.

The honest ledger

What this model shows:

• an image can be represented and learned as a sparse sum of localized Gabor wave-packets (the V1 / Olshausen–Field model, made trainable);
• the phase-wrapping problem is dodged by outputting complex (a, b) coefficients instead of raw angles;
• the correction half of predictive coding works live: a blurry top-down prior is sharpened by gradient descent against a bottom-up residual (the webcam frame).

What it is not, and what it lacks:

• not a photorealistic generator. It is a VAE with an amortized latent — sharp reconstructions, blurry-but-structured samples. It is interesting on biological representation, not on benchmarks against diffusion models;
• the prior is flower-only: in-domain (a flower) the top-down gist genuinely helps; off-domain (a room, a face) it is a wrong guess that the live-frame fit overwrites, so there the loop is doing direct re-fitting, not prediction;
• the "floaters": on out-of-domain input, with an aggressive learning rate, the optimizer orphans some packets into tiny ultra-bright dots instead of coordinating them into edges. They resemble phosphenes, but the cause is an MSE optimizer with no lateral inhibition between packets, not the neural disinhibition that makes real phosphenes — a rhyme, not the mechanism. It points at the missing inhibitory coordination (the grown_gates line), not at a recreated brain;
• it is 2D; relative units throughout; one trained model.

The bet (untouched): that the phase-locked frame is a felt sharpening rather than a computed one. The model locates the mechanism in code that can fail; it does not touch the hard problem.

Lineage

A sub-organ of the_artificial_cortex (the_splat/). The generator is the trained top-down prior; the live cortex is reality forcing that prior into focus. MIT.

The generator dreamed a blurry flower because it had nothing to look at. Open its eyes and reality supplies the phase the dream could not; the packets lock, the gist sharpens, and what they cannot yet coordinate, they see as stars.