Question about your NPC alignment DPO process
Hi, I came across NPCAlign and the Space you built for collecting preference ratings on NPC dialogue. I'm researching how people run preference training on open-weight models. I'm curious how the collected ratings turn into your DPO pairs, and how you figure out whether a run actually improved the dialogue the way you wanted.
Would you be open to a quick chat? Happy to do email if easier, I'm at austin@aureliusaligned.ai.
Just trying to learn, not selling anything.
Thanks either way,
Austin
Hi Austin,
Sorry for the late reply — just noticed this.
Just to clarify the pipeline: the DPO pairs weren't derived from ratings collected in the Space. They were synthetically generated. I used the SFT model to produce two candidate responses per turn (temperatures 0.7 and 1.0), then scored all three candidates (two generated + ground truth) with Gemma 4 26B as an LLM judge on five criteria. Pairs with a score gap ≥ 2/25 were kept as (chosen, rejected), giving 1,341 pairs total.
The Space is for human evaluation — a blind A/B comparison of SFT vs DPO, not for generating training data.
For automated evaluation I used phase-stratified sampling across three conversation phases and measured ROUGE-L, BERTScore-F1, Self-BLEU, and BLEURT. The full pipeline and results are in the README: https://github.com/Hermit888/NPCAlign
Happy to chat further if useful.
Best,
Yuxuan