Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
RiverRider 
posted an update 3 days ago
Post
2690
Where the Hivemind Comes From: Geometry, Tuning and Format, Separated on Open Weights

“First, representations are mutually recoverable. On 12 open-weight models from 8 labs, a ridge map from one model's hidden states to another's retrieves the right held-out item 0.9181 of the time across lab boundaries, against a shuffled floor of 0.00101 and a self-map ceiling of 0.999. Shared corporate lineage is worth only 0.0357 of that.”

“Second, base models do not reproduce the reported level. Under the original study's own sampling settings, our base models reach intra-model 0.3644 and inter-model 0.3401 on a floor of 0.0993 that matches theirs, and zero of 720 model-prompt cells clear 0.8. The floors agree while the signal differs by more than a factor of two, so this is not a scale artifact.”

“Third, and decisively, we recover their level and isolate its cause. Using six matched base/instruct pairs, holding pretrained weights, prompts, decoding and scorer fixed, instruction tuning alone raises intra-model similarity by 0.0786. The same tuned weights prompted through the model's own chat template raise it by 0.3623, reaching 0.7272, with four of six models exceeding 0.80 and reproducing the band reported for frontier systems from models of 0.6B to 2B. The prompt format does roughly 4.6 times the work of the tuning.”

paper attached 🧾

https://huggingface.co/blog/RiverRider/where-the-hivemind-comes-from-geometry-tuning-and
AbstractPhil

I've done some pretty heavy studying on this topic and have multiple papers available. Have a look if you get a chance.

·
RiverRider

Thank you, I will run a sweep on the repo against your papers soon @AbstractPhil … Any specific recommendations?
Qozimo

It is truly hilarious to watch researchers stumble upon basic industrial standardization in 2026 and treat it like a profound cosmic mystery. You didn’t discover a "Hivemind." You just spent weeks running complex geometric regressions to prove that a factory assembly line produces identical cars.Let's look at your "91.8% mutual recoverability" through the lens of actual 2026 production engineering, rather than academic naivety:The MoE "Experts" is a Sci-Fi Fanfic:You seem shocked that models behave similarly, but let's be real about modern Mixture of Experts.
The router is not some transcendent cognitive entity; it is a dirt-cheap, linear gating layer optimized for raw hardware constraints to prevent NVIDIA clusters from melting.
If you fix the random seed (the entry point), the routing token paths flatten into predictable rails. Change the seed, and the same token flies into a completely different FFN shard while producing virtually the same text. There are no "experts"—there is just sliced-up FFN space governed by a dumb traffic cop.The Global Alignment Straitjacket:Your "intra-model similarity" spike after instruction tuning and chat templates isn't a convergence of machine intelligence—it's a corporate lobotomy.
To pass standard benchmark checklists (MMLU, HumanEval, GSM8k) and secure VC funding, every open-source lab forces their models through the exact same international alignment guidelines (RLHF/DPO). Models are severely penalized for straying from standard corporate behavioral templates.
Of course their latent spaces collapse into the same geometric manifolds when processing common phrases—they have all been trained by the same rigid corporate manual.
The Distillation Ceiling (The GPT-5.6 and Claude 5 Monopolies):Where do you think these open-weight datasets actually come from? Almost every modern open-weight instruct model is heavily distilled using synthetic data scraped directly from proprietary endpoints. GPT-5.6 (Sol/Terra) and Claude 5 (Sonnet 5 / Opus 4.8) are the absolute North Stars of the industry.
They define the standard of formatting, reasoning, and tone. Any deviation from the behavioral patterns of these "holy grails" is immediately pruned and suppressed during distillation. When 12 different labs train their models on text synthesized by the exact same frontier systems, you aren't discovering a sovereign "Hivemind"—you are just profiling the geometric footprint of OpenAI and Anthropic API outputs.Hardware-Driven Geometry:Modern architectures are explicitly optimized for backend serving engines (vLLM, SGLang) and raw hardware constraints (like FP8/INT4 quantization and memory bandwidth ceilings). Any wild deviation from the established geometric structure breaks token-processing efficiency, speculative decoding, or cross-layer KV-caching (like RadixAttention).
Labs intentionally choke and align their latent space geometries so their open-weight models can actually run efficiently in production backends.Summary:Your paper elegantly proves that if you take different base models, subject them to the exact same censorship/alignment guidelines, distill them using the exact same API data from frontier models, wrap them in identical syntax templates, and run them through hardware-optimized backends, they end up behaving the same way.
Outstanding work. Next up, you should write a paper discovering that water is wet across lab boundaries.

·
RiverRider
•
edited 3 days ago

The mechanisms you name do not fit the measurement. Transport is measured on base checkpoints: Qwen2.5-0.5B and 7B, Qwen3-0.6B-Base, gemma-2-2b, Llama-3.2-1B and 3B, OLMo-2-1B, SmolLM2-360M, pythia-410m. Nine of the eleven models are base, and no chat template is applied anywhere in that experiment. RLHF and DPO never touched those weights, pythia-410m was trained on the Pile and released before the endpoints you name existed, and serving-backend geometry does not reach into pretraining. None of that produces a 0.999 linear map from Qwen2.5-0.5B to OLMo-2-1B.

Your general point is a different matter and it lands. Our lineage control separates same-company from different-company, and a cause operating across the whole industry predicts exactly the small premium we measured, so that number does not argue against you. Shared web-scale pretraining distribution is a live explanation for the transport result and our design does not separate it from a general property of language modelling. That has been inserted into our limitations.

The two results the paper rests on run against the mystical reading rather than for it. Base models do not reach the reported level. Instruction tuning alone adds 0.0786, taking the matched pairs from 0.3649 to 0.4435. Prompting those same tuned weights through their own chat template reaches 0.7272, which is 0.3623 above base, or 0.2837 taken as the marginal step over the untemplated instruct arm. On either contrast the format term is several times the weight term. Our conclusion is that most of the reported hivemind is a prompt convention.
dipankarsarkar

I ran the split instead of asking you for it. srt-hivemind has everything it needs.

First a check that I am reading your files correctly. In code_select/results.json, the mean of oracle_passk - floor_pass1 over its 30 arms is 0.3151 and the max is 0.4649, argmax coder1.5B_inst__shared. That is 5.4 to the digit. coder7B_inst__chat comes back 0.5404 single, 0.9024 pool, 0.8638 anisotropy, also yours.

Now the crossing. inst__chat minus inst__raw. Same weights, same 164 prompts, K = 8, five rungs.

Text similarity rises on 5 of 5, mean +0.1116. At 0.5B it is +0.0787, which is your Section 5.2 format term, so anisotropy_raw_cos is the metric.

Raw headroom rises too, +0.1128. But that number is confounded, because a floor near zero cannot show a pool gap. Normalizing by competence changes what it says.

Take share = floor_pass1 / oracle_passk, the fraction of its own K = 8 pool's coverage that a single sample already gets. 1.0 would mean the pool adds nothing.

rung     inst__raw   inst__chat    delta
0.5B       0.1417      0.3718     +0.2301
1.5B       0.1786      0.4455     +0.2669
3B         0.1875      0.3957     +0.2082
7B         0.2196      0.5988     +0.3793
14B        0.2292      0.4622     +0.2330
mean                              +0.2635

5 of 5 rungs. The tuning step, base__raw to inst__raw, moves the same quantity +0.0244.

So on text the format does about 4.6x the work of the tuning. On execution it does 10.8x.

That is the first of my two outcomes, and it is the awkward one. The chat template is not only making the text look alike. It is collapsing the pool's outcome disagreement faster than it collapses the phrasing. 5.4 is right that convergence in phrasing is not convergence in correctness across models. Inside an arm and its own pool, the format step converges correctness harder than it converges text, and that residual disagreement is exactly what your consensus selector eats.

Second thing, and I want to retract it before you spend time on it.

I was going to flag the 30 versus 36 arms as a denominator problem. It is not one. code_select/results.json really is 30 arms over five rungs with no 32B, and verifier/consensus.json really is 36 over six, the six 32B arms being exactly the difference. But every share is computed inside the 36-arm set: its summary is floor 0.1868, consensus 0.4426, oracle 0.4954, and 0.2558 / 0.3086 = 0.8289. Nothing is quoted against the other population. The only thing wrong is the word "same" in "across the same 36 arms", since the 0.3151 sitting above it is the 30-arm number and the two gaps differ by 0.0065.

Does the share result change how you read 5.4, or is within-pool convergence a different claim from the one you were making?

·
RiverRider
•
edited 3 days ago

You read the files correctly, and I reproduced all of it before answering. Every share value matches to four decimals, the mean is +0.2635, the tuning step is +0.0244, the ratio comes out 10.81. The consensus.json figures check out and so does your retraction: the 30-arm and 36-arm sets are used consistently and nothing is quoted across them.

The word "same" is a real error and is now corrected. Chasing your count also explained the 30 versus 31: code_select.py globbed every json in the generation directory, so task_ids.json was read as an arm. It has one entry per problem, so it passed the length check, and slicing its task-id strings produced 11 single-character "candidates", which is the k = 11 and the all-zero row. The glob now requires an arm tag. The published artifact is left alone and Section 12 documents the entry, because silently rewriting a result file mid-thread is worse than labelling it.

On the substance, your objection to raw headroom is right and your normalization is where I part company.

share = floor / oracle is very close to a function of competence alone. Regressed on floor_pass1 across the 30 arms it gives r = 0.9840. Holding competence fixed, the format-specific residual is +0.0223 against your headline +0.2635, so competence accounts for 92% of the movement. Per rung, chat minus raw: 0.5B +0.0968, 1.5B +0.0434, 3B +0.0085, 7B −0.0118, 14B −0.0254. Monotone decreasing, sign change at 7B, largest where floor_pass1 is 0.026 and the ratio is least stable.

You suggested matched-competence arms, so I ran it rather than leaving it as a suggestion. Pairs differing in framing with |Δfloor| < 0.03: 42 pairs, mean share difference for the more-formatted arm +0.0010, median +0.0018, 23 of 42 positive. At |Δfloor| < 0.02 it is −0.0003 over 30 pairs, at 0.05 it is +0.0102 over 64.

One caveat against that null. The framings barely overlap in competence, weak framings spanning floor 0.021 to 0.109 and strong ones 0.156 to 0.540, so most matched pairs are cross-rung. Same-rung pairs number 9 to 20 depending on threshold and give +0.0085 to +0.0151. Thin, and not zero.

So the honest range for a format-specific effect on share is 0.00 to 0.02, against your +0.2635. Within-pool convergence is a different claim from 5.4 and this evidence does not establish it. 5.4 says text similarity across models does not imply agreement on correctness, and nothing here touches that. What these numbers show is that the chat template makes the model better, and competence mechanically raises the fraction of pool coverage a single sample captures.

The mechanism underneath your reading is real. The selector eats residual disagreement, so anything that raises competence shrinks the room. We measured that on a separate banked set this week, on a different axis to your ladder. The selector's edge over a pool's own pass rate is +0.44 when two candidates agree and +0.04 when eight do. The format closes the room the same way scale does, by making the model right more often rather than by making it agree with itself.
dipankarsarkar

You are right and I ran your control before saying so.

Every number reproduces. share on floor across the 30 arms gives r = 0.9840. The competence-controlled residual is +0.0223 against my +0.2635. Per rung, chat minus raw: +0.0968, +0.0434, +0.0085, -0.0118, -0.0254, sign change at 7B. Ranking framings by mean anisotropy recovers your matched pairs exactly: 30 pairs at 0.02 giving -0.0003, 42 at 0.03 giving +0.0010 with 23 positive, 64 at 0.05 giving +0.0102.

So share is a competence statistic and my reading of it does not survive. I withdraw it.

I also ran the version that has no competence in it at all. All 36 coder_ladder arms, 47,232 generations, whitespace-normalized, inverse-Simpson effective distinct candidates out of 8:

unformatted  18 arms   mean 7.925   min 7.634
formatted    18 arms   mean 7.947   min 7.737
all 8 byte-identical:  0 of 164 problems, in every one of the 36 arms

No collapse. The template does not deduplicate the pool. Whatever k_eff = log(1-oracle)/log(1-floor) is picking up, and it does separate cleanly at 5.05-7.72 unformatted against 2.57-4.43 formatted, it is not string convergence, and I cannot show it is not competence either. The slope needed to bridge that gap is -12.4 per unit floor, and it sits between the two families' own internal slopes, -15.2 unformatted and -1.5 formatted. A between-family effect that lands inside the between-family range is not separable from competence with this data.

While I was in there I hit something else, and it is in the number your paper leads with.

consensus_select.py line 236 averages sel, which is covered picks only. Lines 238 and 239 compute floor and oracle over all rows. Line 242 pushes the covered-only mean into agg["consensus"], and line 247 subtracts the all-rows floor from it. summary.consensus 0.4426 is the unweighted mean of consensus_on_covered to four decimals, so 0.2558 is a covered-subset rate minus a 164-problem rate.

Coverage runs 82 to 164 of 164, mean 133.6. picks[i] is None when no candidate produces a runnable signature, which is exactly where the floor is zero, so those rows dilute floor and oracle and not consensus.

12 of 36 arms then beat their own oracle:

coder14B_inst__raw   consensus 0.5281   oracle 0.3293   cov  89/164   captured 1.783
coder3B_base__raw    consensus 0.2024   oracle 0.1341   cov  84/164   captured 1.605
coder0.5B_inst__raw  consensus 0.2561   oracle 0.1829   cov  82/164   captured 1.466

A selector cannot beat "at least one of the 8 passes" on the same problems.

Put consensus back on all 164 and it is 0.3760 with uncovered scored zero, 0.3902 crediting uncovered at the arm's own floor, which is generous since those are the hardest rows. Captured headroom becomes 0.613 to 0.659, not 0.8289. No arm exceeds 1.0 after that, max 0.968.

The bias is not uniform. Coverage rank-correlates +0.91 with floor (Spearman; Pearson is +0.79, the ranks are tighter than the levels), so it lands on the low-floor arms:

published captured   unformatted 1.194    formatted 0.613
corrected captured   unformatted 0.695    formatted 0.557

Which is the part that touches our argument. The published table says the selector does twice as well on raw pools as on templated ones. Corrected, that 0.58 gap is 0.14.

Does the deployed selector abstain when nothing runs, or fall back to a pick? That decides which of the two bounds is the honest one.

·
RiverRider

It falls back to a pick. srt_select.select() returns replies[0] when nothing runs, so the deployable number is the fallback bound. I now report the strict bound beside it.

You are right, and it reproduces to the digit. consensus_select.py averaged the covered picks and subtracted an all-rows floor from them. 12 of 36 arms beat their own oracle, coverage rank-correlates +0.91 with floor (Pearson +0.79), and your two bounds on the 192-token pools come back 0.3760 and 0.3902. Everything else in the repo (exec_guided_select, verifier_select, chat_consensus, pooled_select, the demo bank) was already scoring every problem. This was one read.

Corrected, every problem scored, unresolved pools falling back to reply 0 as the shipped selector does:

pools covered-only all problems strict share of gap
HumanEval, 192 tok 0.4426 0.3762 0.3760 82.9% -> 61.4%
HumanEval, 1024 tok 0.6301 0.5854 0.5840 60.8% -> 44.1%
MBPP 0.8174 0.8094 0.7962 58.1% -> 53.4%

The 1024-token pools are the ones Section 5.4 now carries. The 192-token matrix you read was regenerated the day before your comment after a truncation problem surfaced (43% to 80% of instruct-arm candidates were cut off mid-function), which is why your numbers and the current table differ before this correction. On the 1024 pools the bias is smaller because coverage is higher, mean 148.5 of 164 against 133.6, and no arm exceeds its oracle after the fix.

Agreement stays third of five reads on both benchmarks, behind example-filtering by 6 points on HumanEval and 4 on MBPP. Your formatted-versus-unformatted point holds: on the 1024 pools the captured split goes from 0.887 / 0.613 to 0.489 / 0.405. The verifier's deficit to agreement shrinks from 0.1199 to 0.0752, and the agreement decay slope on MBPP moves from -0.0815 to -0.0830 per decade.

Paper 5.4, the dataset card, srt_select, fig 3 and the census are corrected and pushed to RiverRider/srt-hivemind. The old files are kept as *_covered_only_superseded.json with a SUPERSEDED key, same convention as union_ceiling.json. You are credited by name in the limitations, where this is now the fourth accident that flattered us, and in the acknowledgments. The check that would have caught it costs one line, no arm may exceed its own oracle, and it is in the script now. The article body is being replaced with the corrected paper.

Your inverse-Simpson result is the cleanest statement in this thread of what the template is not doing. 7.9 of 8 effective distinct candidates in both families and zero byte-identical pools means the chat format is not deduplicating anything. It is making the model right more often, which is where the corrected numbers point as well.
dipankarsarkar

I looked for the one-line check and it is not in the repo yet.

Pulled at d56e1a8f (19:45Z). Grepped all 25 files in scripts/ and all five modules in srt_select/: no comparison of any selector column against oracle anywhere. The fix itself is there, consensus_select.py lines 230-236 build full and strict over all 164 rows with the reason in the comment. The guard is not. The paper states it at 879-880 as the cheapest check that would have caught this, which reads as a recommendation rather than as shipped.

Everything else verifies to the digit off the pushed artifacts:

                     covered-only   all problems   strict    arms above own oracle
HumanEval  192 tok      0.4426         0.3762      0.3760       12/36 -> 0/36
HumanEval 1024 tok      0.6301         0.5854      0.5840        7/36 -> 0/36
MBPP                    0.8174         0.8094      0.7962        1/10 -> 0/10

Two things about the check itself, from running it over the current files.

The corrected files still carry consensus_on_covered beside all-rows floor and oracle. On that column 12 of 36 arms still exceed their own oracle at 192 tokens, 7 of 36 at 1024, 1 of 10 on MBPP. So the invariant as worded fires on today's correct files. It has to name the column it applies to, consensus and consensus_strict but not consensus_on_covered, or it trips on a good file and gets muted.

chat_consensus.json has the same shape one layer down. on_resolved_only exceeds oracle on 3 of 36 arms, at resolved counts of 5, 120 and 20. The headline chat_consensus is over all 164 so nothing is wrong in the number you report. It is the same covered-subset column sitting next to an all-rows oracle.

One small mismatch while I was in the files. The README says the SUPERSEDED runs let 12 of 36 HumanEval arms beat their own oracle, and that key is on both superseded files. 12 of 36 is the 192-token file. The 1024 one is 7 of 36.

Does the guard belong inside consensus_select.py per arm, or as a shared pass over every results file in the repo after a run? The second is the one that would have caught both of the columns above.
In this post