Papers
arxiv:2310.13017
Position Interpolation Improves ALiBi Extrapolation
Published on Oct 18, 2023
Authors:
Abstract
Linear position interpolation enhances the extrapolation capability of models using Attention with Linear Biases and rotary position embeddings for language modeling and downstream tasks.
Generated by Qwen/Qwen2.5-Coder-32B-Instruct
Linear position interpolation helps pre-trained models using rotary position embeddings (RoPE) to extrapolate to longer sequence lengths. We propose using linear position interpolation to extend the extrapolation range of models using Attention with Linear Biases (ALiBi). We find position interpolation significantly improves extrapolation capability on upstream language modelling and downstream summarization and retrieval tasks.
Get this paper in your agent:
hf papers read 2310.13017 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 3
Datasets citing this paper 0
No dataset linking this paper
Cite arxiv.org/abs/2310.13017 in a dataset README.md to link it from this page.
Spaces citing this paper 18
Browse 18 spaces citing this paperCollections including this paper 0
No Collection including this paper
Add this paper to a collection to link it from this page.