| --- |
| license: apache-2.0 |
| language: |
| - es |
| - en |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - markdown |
| - text-to-markdown |
| - faithful-generation |
| - llama |
| - from-scratch |
| - spanish |
| - english |
| - ocr-postprocessing |
| - rag |
| - document-understanding |
| - tiny-llm |
| model-index: |
| - name: PASITA |
| results: |
| - task: |
| type: text-generation |
| dataset: |
| name: PASITA-bench-1000 (held-out, private) |
| type: custom |
| metrics: |
| - name: gfm_validity |
| type: accuracy |
| value: 0.956 |
| - name: faithfulness |
| type: faithfulness |
| value: 0.927 |
| - name: semantic_faithfulness |
| type: faithfulness |
| value: 0.888 |
| - name: table_fidelity |
| type: accuracy |
| value: 1.0 |
| --- |
| |
| # PASITA v1 — plain text to Markdown (faithful) |
|
|
| A **decoder-only language model trained 100% from scratch** (no base model), specialized in a single task: converting **plain text into valid Markdown while preserving information**. |
|
|
| > Compiler behavior, not chatbot behavior: it adds structure, it does not invent content. |
|
|
| ## Architecture (lab notes) |
|
|
| | Parameter | Value | |
| |---|---| |
| | Class | `LlamaForCausalLM` (decoder-only, dense, no MoE) | |
| | Parameters | **88,099,584** (~88M) in `bfloat16` | |
| | Layers / hidden / FFN | 12 / 768 / 2048 (SwiGLU) | |
| | Attention | GQA 12Q/4KV, head_dim 64, no bias | |
| | Positions | RoPE theta=100000, ctx 2048, RMSNorm eps=1e-5 | |
| | Embeddings | tied (saves ~12.6M params) | |
| | File | `model.safetensors` (176 MB, 110 tensors, sha `664665e7…`) | |
| | Tokenizer | Custom 16k byte-level BPE, verified ByteLevel decoder (~4.0 chars/token ES/EN) | |
| | Special tokens | `<pad> <s> </s> <unk> <think> </think>` | |
| |
| ## Training |
| |
| 1. **SFT** 58M tokens x2 epochs — final loss 0.09, token accuracy 98.4% |
| 2. **DPO** beta=0.1 — preference margin 4.2 |
| 3. **GRPO** 550+150 steps, G=4, verifiable rewards (format + numeric fidelity + anti-overformatting) |
| 4. Data: 80M human markdown-derived tokens (ES/EN Wikipedia, StackExchange, WikiHow) scaled to 625M in v4 corpus |
| |
| ## Benchmark (held-out n=1000, greedy) |
| |
| | Global | GFM 0.956 - faith 0.927 - sem 0.888 - tables 1.0 | |
| |---|---| |
| | code / ocr / docs / math / tables / html | 0.93 - 1.00 | |
| | control (strict instructions) | 0.19 (known limitation) | |
| |
| ## Usage |
| |
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| tok = AutoTokenizer.from_pretrained("OpceanAI/PASITA") |
| model = AutoModelForCausalLM.from_pretrained("OpceanAI/PASITA", dtype="auto") |
| prompt = "CONVIERTE A MARKDOWN:\n" + text + "\n\n### Markdown:\n" |
| ids = tok(prompt, return_tensors="pt", truncation=True, max_length=1024) |
| out = model.generate(**ids, max_new_tokens=512, do_sample=False) |
| print(tok.decode(out[0][ids.input_ids.shape[1]:], skip_special_tokens=True)) |
| ``` |
| |
| Valid regime: medium/long documents (OCR output, pasted HTML, meeting notes, tutorials). Fragile on 1-3 line inputs. |
| |
| ## Limitations |
| |
| May truncate digits, drop secondary data, emit echo H1s, or continue past completion. Recommended: adaptive `max_new_tokens` + beam search + fidelity rerank. |
| |
