File size: 3,030 Bytes
6753a8a 262104e 6753a8a 4adddc8 6753a8a 4adddc8 6753a8a 4adddc8 6753a8a 4adddc8 6753a8a 4adddc8 262104e 4adddc8 6753a8a 4adddc8 6753a8a 4adddc8 6753a8a 262104e 6753a8a 4adddc8 262104e 4adddc8 6753a8a 4adddc8 6753a8a 262104e 4adddc8 262104e 6753a8a 4adddc8 6753a8a 4adddc8 6753a8a 4adddc8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 | ---
license: apache-2.0
language:
- es
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- markdown
- text-to-markdown
- faithful-generation
- llama
- from-scratch
- spanish
- english
- ocr-postprocessing
- rag
- document-understanding
- tiny-llm
model-index:
- name: PASITA
results:
- task:
type: text-generation
dataset:
name: PASITA-bench-1000 (held-out, private)
type: custom
metrics:
- name: gfm_validity
type: accuracy
value: 0.956
- name: faithfulness
type: faithfulness
value: 0.927
- name: semantic_faithfulness
type: faithfulness
value: 0.888
- name: table_fidelity
type: accuracy
value: 1.0
---
# PASITA v1 — plain text to Markdown (faithful)
A **decoder-only language model trained 100% from scratch** (no base model), specialized in a single task: converting **plain text into valid Markdown while preserving information**.
> Compiler behavior, not chatbot behavior: it adds structure, it does not invent content.
## Architecture (lab notes)
| Parameter | Value |
|---|---|
| Class | `LlamaForCausalLM` (decoder-only, dense, no MoE) |
| Parameters | **88,099,584** (~88M) in `bfloat16` |
| Layers / hidden / FFN | 12 / 768 / 2048 (SwiGLU) |
| Attention | GQA 12Q/4KV, head_dim 64, no bias |
| Positions | RoPE theta=100000, ctx 2048, RMSNorm eps=1e-5 |
| Embeddings | tied (saves ~12.6M params) |
| File | `model.safetensors` (176 MB, 110 tensors, sha `664665e7…`) |
| Tokenizer | Custom 16k byte-level BPE, verified ByteLevel decoder (~4.0 chars/token ES/EN) |
| Special tokens | `<pad> <s> </s> <unk> <think> </think>` |
## Training
1. **SFT** 58M tokens x2 epochs — final loss 0.09, token accuracy 98.4%
2. **DPO** beta=0.1 — preference margin 4.2
3. **GRPO** 550+150 steps, G=4, verifiable rewards (format + numeric fidelity + anti-overformatting)
4. Data: 80M human markdown-derived tokens (ES/EN Wikipedia, StackExchange, WikiHow) scaled to 625M in v4 corpus
## Benchmark (held-out n=1000, greedy)
| Global | GFM 0.956 - faith 0.927 - sem 0.888 - tables 1.0 |
|---|---|
| code / ocr / docs / math / tables / html | 0.93 - 1.00 |
| control (strict instructions) | 0.19 (known limitation) |
## Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("OpceanAI/PASITA")
model = AutoModelForCausalLM.from_pretrained("OpceanAI/PASITA", dtype="auto")
prompt = "CONVIERTE A MARKDOWN:\n" + text + "\n\n### Markdown:\n"
ids = tok(prompt, return_tensors="pt", truncation=True, max_length=1024)
out = model.generate(**ids, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True))
```
Valid regime: medium/long documents (OCR output, pasted HTML, meeting notes, tutorials). Fragile on 1-3 line inputs.
## Limitations
May truncate digits, drop secondary data, emit echo H1s, or continue past completion. Recommended: adaptive `max_new_tokens` + beam search + fidelity rerank.
|