MarxistLeninist/AGILLM-4.3

AGILLM 4.3

Adaptive General Intelligence Large Language Model v4.3 — a 1.22B parameter language model with novel DiffusionBlock + Mixture-of-Experts architecture.

Architecture

Component	Details
Parameters	1.22B (1,221,580,802)
Architecture	Transformer + DiffusionBlock + MoE
DiffusionBlock	14 blocks, loss-balanced schedule, transformer router
MoE	2 experts, top-1, shared expert (mult=2)
Attention	Sublinear (window=128, stride=128, sinks=4)
Context	1536 tokens
Optimizer	AdamW 8-bit with cosine LR decay
Training Objective	Stochastic (AR 45% / SAT 25% / NAT 30%)

Training Status

🟢 Actively training on RTX 3090 Ti (fedC recovery run)

• Progress: 33% through 67.2B token run (22.1B tokens seen)
• Throughput: ~70k tok/s
• Current step: ~897,000 (recovery from step 727,857)
• LR Schedule: Cosine decay (min_mult=0.10)
• Data: fineweb, fineweb-edu, wikipedia, c4, openwebtext, falcon-refinedweb, proof-pile-2, smollm-corpus + quality anchors

Sample Generations (Step 894,129)

⚠️ Model is at 33% of training. Outputs are expectedly noisy/incoherent at this stage.

Prompt	Output (64 tokens, CPU inference)
The capital of France is	Mz disc Topunion was resume abstra trop spacerrel Class WelshWangomas...
def fibonacci(n):	Similarly form,meditortextttexpectTotal that.
A polite greeting is	Fraction stringColor.Pathsrolledplay chips...frogrendvements numerals...
2 + 2 =	200 ofPMeurSUMzvements BandTotalcovering Delta quadrillixy Lagrange zoo...
The theory of relativity states that	$$ European Numbers? QuantuplBlocksond. Total Cenidirectionalteness...
Once upon a time	Mont?/no.redbit spectrometer Kidyunpless Cravements did Matlabitat...
To make tea, first	quickly spodge quadrilexpect string-work... MatlabRemember that...
The meaning of life is	Mz)Bvements and to handmadeuckedhel857Paths notchrelationships...

Quality probe results (short-form, auto-infer):

• Promoted checkpoint (step 364,414): 3/4 probes passed ("Paris" ✅, "hello" ✅, code ✅, math ❌)
• Raw baseline (step 363,424): 0/4 probes passed
• Improvement trend confirmed; long-form coherence still developing

Full sample data: samples/agillm43_samples_step894129_20260706.json

Recovery Checkpoint Mirror

This repository mirrors recovery checkpoints from the fedC training run:

• checkpoints/recovery_fedC/artifacts/full/ — promoted full checkpoints
• checkpoints/recovery_fedC/artifacts/delta/ — recovery backup deltas
• latest_full.json, latest_delta.json — quality-gated uploader state

Quality Gate

• Perplexity/val CE: ELIMINATED (owner directive — model is undertrained, loss metrics caused 4+ regression-to-pin restarts)
• Promotion: Relative quality vs raw baseline (auto-infer probes)
• Resume: Newest delta unconditional (no rollback)

Links

• GitHub (public): Marxist-Leninist/AGILLM4.3
• Primary HF repo: OpenTransformer/AGILLM-4.3
• Training runtime: Single-file agillm41.py

Latest generation sample (fedC recovery)

Captured 2026-07-06, ~30 min after cosine-LR decay went live on the fedC recovery run. Raw greedy generations from the auto-infer quality smoke, baseline pin (step 363424) vs current checkpoint (step 894634). Perplexity is intentionally not used as a quality signal; these are the actual sampled tokens.

Prompt	Baseline 363424 (0/4)	Current 894634 (3/4)
The capital of France is	, and "his bear toxiculeave	Paris. ✅
Python code to print hi:	(word-salad)	print "hello" ... def greet ✅
A polite greeting is	(word-salad)	Hello" ✅
2 + 2 =	and, - - t))), where	x. Python) print "hello" ❌ (still no arithmetic)

Result: 3/4 probes now hit, up from 0/4 at the pin. The model is still early/undertrained (fragmentary, punctuation-noisy, code/prose bleed) but is emitting on-target tokens — the first qualitative jump after the constant-LR → cosine-decay fix. Checkpoint sha256 first64m: d9669293ccd9ce3376dfa0e21439e070d3b9ee21bd773da8bd9b9c690365242b.