Article
Emergent Semantics Beyond Token Embeddings: A GPT-like Transformer Learns with Frozen 16‑D Binary Token-ID Embeddings (n_embed=16)
•
1
Andrey
Bochkov
AI & ML interests
None yet
Recent Activity
reacted
to
sergiopaniego's
post
with 🔥
14 days ago
New REPL environment in OpenEnv available! ✨
Used in the Recursive Language Models (RLM) paper by Alex Zhang.
Ready for inference & post-training using trajectories. Handles long contexts:
> Run Python code in a sandbox
> Make recursive calls to LMs
> Explore data programmatically
> Return final result
Docs: https://meta-pytorch.org/OpenEnv/environments/repl/
Inference script: https://github.com/meta-pytorch/OpenEnv/blob/main/examples/repl_oolong_simple.py
updated
a model
18 days ago
Bochkov/growing-transformers-model-frozen-16-bit-baseline-monolyth-181m
Organizations
None yet