R PRO

The entire fingerprint of the model fits in a Hugging Face comment

model.layers.0.self_attn.q_proj.weight                 -0.000429   0.704143   0.0000      n/a
  model.layers.0.self_attn.k_proj.weight                  0.000637   0.681670   0.0000      n/a
  model.layers.0.self_attn.v_proj.weight                 -0.000270   0.584485   0.0000      n/a
  model.layers.0.self_attn.o_proj.weight                  0.000146   0.590797   0.0000      n/a
  model.layers.0.mlp.gate_proj.weight                    -0.002121   0.660124   0.0000      n/a
  model.layers.0.mlp.up_proj.weight                       0.000017   0.639064   0.0000      n/a
  model.layers.0.mlp.down_proj.weight                    -0.000128   0.674867   0.0000      n/a
  model.layers.1.self_attn.q_proj.weight                  0.000126   0.460640   0.0000      n/a
  model.layers.1.self_attn.k_proj.weight                 -0.000302   0.497939   0.0000      n/a
  model.layers.1.self_attn.v_proj.weight                 -0.000028   0.401955   0.0000      n/a
  model.layers.1.self_attn.o_proj.weight                 -0.000174   0.389998   0.0000      n/a
  model.layers.1.mlp.gate_proj.weight                     0.000997   0.347956   0.0000      n/a
  model.layers.1.mlp.up_proj.weight                       0.000004   0.294881   0.0000      n/a
  model.layers.1.mlp.down_proj.weight                     0.000024   0.351096   0.0000      n/a
  model.layers.2.self_attn.q_proj.weight                 -0.000131   0.581838   0.0000      n/a
  model.layers.2.self_attn.k_proj.weight                  0.000281   0.632862   0.0000      n/a
  model.layers.2.self_attn.v_proj.weight                 -0.000222   0.835755   0.0000      n/a
  model.layers.2.self_attn.o_proj.weight                  0.000309   0.658466   0.0000      n/a
  model.layers.2.mlp.gate_proj.weight                    -0.000002   0.430325   0.0000      n/a
  model.layers.2.mlp.up_proj.weight                      -0.000071   0.370586   0.0000      n/a
  model.layers.2.mlp.down_proj.weight                     0.000095   0.430278   0.0000      n/a
  model.layers.3.self_attn.q_proj.weight                 -0.000144   0.603692   0.0000      n/a
  model.layers.3.self_attn.k_proj.weight                  0.000003   0.663869   0.0000      n/a
  model.layers.3.self_attn.v_proj.weight                  0.000626   1.357608   0.0000      n/a
  model.layers.3.self_attn.o_proj.weight                  0.000072   0.863481   0.0000      n/a
  model.layers.3.mlp.gate_proj.weight                     0.000673   0.408003   0.0000      n/a
  model.layers.3.mlp.up_proj.weight                      -0.000106   0.381201   0.0000      n/a
  model.layers.3.mlp.down_proj.weight                    -0.000099   0.378011   0.0000      n/a

I would have guessed that reintroducing tensors which originated from a non-abliterated variant would have had a negative impact on refusals. How ever it would make sense that the replacement of a targeted and well managed area of vector repairs makes perfect sense. The refusal mechanisms don't regain persistence with the reintroduction of repaired vectors due to the fact that the refusal weight doesn't have the support it needs to activate. Without similar neighbouring weights it's just an alarm without a power source to complete its circuit.

When you are repairing a model from excessive abliteration damage which methods of vector replacement are you using? SLERP would make sense at a low ratio but could TIES be effective as well? Have you found a replacement ratio that has allowed the refusal circuit to regain its dominance allowing the baseline refusals to revert?

This idea has definitely got my mind racing with new possibilities for my research projects. I plan on trying post abliteration repair out this evening. This also has me thinking of hybridizing the training pipeline. I would like to try an SFT with fewer epochs than I would apply to a fully trained model, followed by abliteration and vector correction and finish the model with 1 more epoch of SFT. You could also substitute the optimizer out and see if adamw vs muon has any effect on refusal.

Call me crazy, but I always thought of how much more efficient it made me in token usage and how much of my work I was actively retaining between sessions. Was it annoying? Absolutely. Did the benefits outweigh the tokens lost? After awhile maybe?