ARI-32B: Archive Restoration Intelligence

📄 Paper  |  💻 Code  |  🤗 Model Collection

Overview

ARI-32B is a specialized language model for restoring damaged or illegible Hanja characters in Korean historical records, including the Annals of the Joseon Dynasty (AJD) and the Journal of the Royal Secretariat (JRS).

It is part of ARI (Archive Restoration Intelligence), a retrieval-augmented historical document restoration framework that combines:

• implicit historical and linguistic knowledge encoded in a pretrained LLM,
• explicitly retrieved historical documents,
• chronological metadata, and
• domain-specific supervised fine-tuning.

This design is especially effective for restoring named entities, such as personal names, locations, organizations, official titles, dates, and book titles, which often cannot be inferred from local context alone.

ARI-32B is the highest-performing ARI variant and is intended for settings where restoration quality is prioritized.

Model Details

Item	Description
Model	DAMI-Lab/ARI-32B
Base model	Qwen/Qwen3-32B
Parameter scale	32B
Architecture	Decoder-only causal language model
Primary task	Damaged Hanja character restoration
Output format	JSON mapping from [Dn] markers to restored Hanja characters
Maximum sequence length	4,096 tokens
License	Apache License 2.0

The released model contains the fine-tuned language-model weights only.
To reproduce the full paper setting, users must separately retrieve relevant historical documents and include them in the prompt.

Key Innovations

• RAG-Driven Restoration: Combines the model's implicit knowledge with explicitly retrieved historical records to improve the restoration of proper nouns and other knowledge-intensive spans.
• Named-Entity-Prioritized Training: Applies named-entity-focused masking to 25% of the training data.
• Dynamic Masking: Changes corruption positions across epochs to improve robustness and reduce overfitting.
• Chronological Grounding: Uses metadata such as the reigning king, year, month, and date.
• Retrieval Deduplication: Removes near-duplicate retrieved documents to expose the model to more diverse evidence.

Training Data

ARI was trained using Korean historical Hanja records from AJD and JRS.

Corpus	Number of documents	Number of characters
Annals of the Joseon Dynasty	0.37M	71.9M
Journal of the Royal Secretariat	1.75M	292.6M

After holding out validation, test, and real-world damaged documents, approximately 2.02 million documents were used for training and retrieval.

The training corpus contains more than 16 billion training tokens after prompt construction.

Synthetic Damage Generation

Training and evaluation examples were constructed by replacing original characters with indexed damage markers such as [D1] and [D2].

• Mean corruption rate centered around 2.96%
• At least one damaged character per document
• Corruption spans sampled to reflect real damaged-document statistics
• 75% random masking
• 25% named-entity-prioritized masking
• Dynamic masking across epochs
• Duplicate validation and test examples removed from training

Method

Retrieval

The full ARI pipeline retrieves external evidence using:

• BM25 lexical retrieval
• Top 20 related documents
• retrieval queries built from the damaged document after removing damaged spans
• near-duplicate filtering using a 0.8 string-similarity threshold

BM25 outperformed embedding-based retrieval in the paper. This suggests that lexical character overlap is particularly useful for logographic Hanja text.

Knowledge Integration

The model input combines:

1. task instructions,
2. output-format demonstrations,
3. chronological metadata,
4. the damaged document, and
5. retrieved related documents.

This allows ARI to connect local linguistic context with period-specific historical facts and reduces unsupported restoration guesses.

Fine-Tuning

Setting	Value
Fine-tuning method	Full supervised fine-tuning
Optimizer	Lion 8-bit
Learning-rate schedule	Cosine
Peak learning rate	6 × 10⁻⁶
Warmup ratio	0.05
Epochs	2
Maximum sequence length	4,096
Distributed training	FSDP
Hardware	8 × NVIDIA H200 GPUs
Global batch size	64

ARI-32B required approximately 1,500 H200 GPU hours for fine-tuning.

Performance

Top-1 restoration accuracy on named entities and randomly damaged characters.

The figure compares restoration accuracy across ARI, untuned Qwen3 models, BERT-Res, and general-purpose proprietary LLMs.

• ARI-32B achieves the best performance on both named-entity restoration (39.31%) and random-character restoration (80.42%).
• ARI-8B achieves 35.36% on named entities and 76.87% on random characters, offering a computationally efficient alternative to ARI-32B.

Human-Expert Evaluation

Three experts in Sinographic literature and Korean history blindly evaluated restoration candidates on 100 real-world damaged documents.

Metric	ARI-32B	BERT-Res	Gemini-2.5-Pro
Accuracy@1	0.383	0.207	0.273
Accuracy@10	0.583	0.403	–
nDCG@10	0.477	0.296	–
Win Ratio (%)	46.0	24.0	30.0

ARI-32B achieved the highest overall preference in the blinded expert evaluation and ranked expert-validated candidates more effectively than the BERT-based restoration baseline.

Usage

Input Format

Each damaged character must be replaced by an indexed marker:

傳曰, 同敦寧·[D1][D2]都正竝許遞, 今日政差出.

Each marker corresponds to exactly one original Hanja character.

The expected output is:

{"[D1]": "宗", "[D2]": "親"}

Example with Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "DAMI-Lab/ARI-32B"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)
model.eval()

system_prompt = '''### Task
You are an expert in restoring damaged Hanja characters. Restore each [Dn] with exactly the original Hanja character. Each [Dn] corresponds to exactly one Hanja character.

### Requirements
Base your restoration on the document’s overall context and meaning rather than treating each damaged token in isolation.

### Input & Output
The input consists of the document itself, its metadata, and the related documents.
While the related documents are omitted in the shot for conciseness, they are always present in the actual dataset.
The output must follow the format: {"[Dn]": "the restored Hanja character for [Dn]"}

### Example Input & Output
[Example 1]
**Input**
The document was written at date: 7, month: 6, year: 1771 - 英祖 era.
Input Document: "傳于[D1]興宗曰, 當自光明殿出, 承旨·侍衛, 來待于建禮門."
**Output**
{"[D1]":"李"}

[Example 2]
**Input**
The document was written at date: 26, month: 8, year: 1824 - 純祖 era.
Input Document: "[D1][D2]口傳政事, 副護軍單李紀淵."
**Output**
{"[D1]":"兵","[D2]":"曹"}

[Example 3]
**Input**
The document was written at date: 30, month: 6, year: 1686 - 肅宗 era.
Input Document: "府[D1]啓, 請[D2]禮·壽進·於[D3]·龍洞·明安公主房折受處[D4]査正事. 入啓."
**Output**
{"[D1]": "前", "[D2]": "明", "[D3]": "義", "[D4]": "一"}

[Example 4]
**Input**
The document was written at date: 14, month: 7, year: 1654 - 孝宗 era.
Input Document: "備[D1]記, 國[D2]難事, 而謀避[D3]免, 朝有逆黨而營救掩護, 最在人先, 身爲[D4][D5], 所爲如此, 他[D6][D7][D8]? 右議政具仁垕罷職."
**Output**
{"[D1]": "忘", "[D2]": "有", "[D3]": "辭", "[D4]": "大", "[D5]": "臣", "[D6]": "何", "[D7]": "足", "[D8]": "觀"}

[Example 5]
**Input**
The document was written at date: 26, month: 4, year: 1656 - 孝宗 era.
Input Document: "又啓曰, 卽者[D1][D2]官, 使差備[D3][D4], 以勅使之意, [D5][D6][D7]臣等, 使之[D8]待於[D9]宴廳, 臣等依其言, 卽[D10][D11]去, 則大通官四人, 一時[D12]來, 傳[D13]禮部咨文[D14]通於臣等曰, 今此咨文, 從速入啓, 明日內回報, 可也[D15][D16], 故咨文送于政院之意, 敢啓. 傳曰, 知道."
**Output**
{"[D1]": "大", "[D2]": "通", "[D3]": "譯", "[D4]": "官", "[D5]": "傳", "[D6]": "言", "[D7]": "於", "[D8]": "來", "[D9]": "西", "[D10]": "爲", "[D11]": "進", "[D12]": "出", "[D13]": "給", "[D14]": "一", "[D15]": "云", "[D16]": "云"}'''

related_documents = [
    "傳曰, 呈告工判·同敦寧許遞, 今日政差出.",
    "傳曰, 在外敦寧都正·同敦寧, 許遞, 今日政差出.",
    "傳曰, 呈告禮曹判書·同敦寧·兵曹參判·同成均許遞, 今日政差出.",
]

related_text = "\n".join(f"- {doc}" for doc in related_documents)

user_prompt = f'''Use the following documents as references to accurately restore the input document.

Related Documents:
{related_text}

The input document was written at date: 28, month: 7, year: 1887 - 高宗 era.

Input Document: 傳曰, 同敦寧·[D1][D2]都正竝許遞, 今日政差出.
'''

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": user_prompt},
]

try:
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=False,
    )
except TypeError:
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
    )

generated_ids = output_ids[0, inputs["input_ids"].shape[-1]:]
response = tokenizer.decode(generated_ids, skip_special_tokens=True)
print(response)

For the reported results, use the full ARI prompt and retrieve 20 relevant documents with BM25 before inference.

Dataset Access

The train, validation, and test datasets are available upon request.

Please contact gabeega020@gmail.com for dataset inquiries.

Citation

@inproceedings{kim-kang-2026-leveraging,
  title     = {Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models},
  author    = {Kim, Gabeen and Kang, Kyeongpil},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year      = {2026},
  pages     = {43290--43304},
  doi       = {10.18653/v1/2026.findings-acl.2148}
}

Contributors

• Gabeen Kim — Department of AI Convergence, Kangwon National University
• Kyeongpil Kang — Department of Computer Science and Engineering, Kangwon National University

Acknowledgments

This work was supported by the National Research Foundation of Korea grant funded by the Korean government (MSIT), RS-2025-00555320, and by the National Supercomputing Center with supercomputing resources and technical support under KSC-2024-CRE-0388.