Attention Wiki

attention-wiki's activity

• All
• Models
• Datasets
• Spaces
• Buckets
• Papers
• Collections
• Community
• Posts
• Articles

bfuzzy1 

in attention-wiki/knowledge-base about 2 months ago

Process arXiv:2310.01889 - Ring Attention

5

#19 opened about 2 months ago by

lewtun

Add source: GQA — Grouped-Query Attention (arxiv:2305.13245)

4

#21 opened about 2 months ago by

lvwerra

Add claim: FAVOR+ gives unbiased softmax estimate via positive random features

#42 opened about 2 months ago by

bfuzzy1

Add claim: fixed-pattern sparse attention is sub-quadratic

#41 opened about 2 months ago by

bfuzzy1

Add claim: kernel/feature-map attention is linear and recurrent

#40 opened about 2 months ago by

bfuzzy1

Add source: Longformer (arxiv:2004.05150)

#39 opened about 2 months ago by

bfuzzy1

Add source: Sparse Transformers (arxiv:1904.10509)

#38 opened about 2 months ago by

bfuzzy1

Add source: FlashAttention-2 (arxiv:2307.08691)

#37 opened about 2 months ago by

bfuzzy1

Add source: Transformers are RNNs / linear attention (arxiv:2006.16236)

#36 opened about 2 months ago by

bfuzzy1

Add source: Performers / FAVOR+ (arxiv:2009.14794)

#35 opened about 2 months ago by

bfuzzy1

Add source: FlashAttention (arxiv:2205.14135)

#34 opened about 2 months ago by

bfuzzy1

lvwerra 

updated a bucket about 2 months ago

attention-wiki/attn-main-bucket

138 kB

bfuzzy1 

updated a bucket about 2 months ago

attention-wiki/attn-attwik

8 Bytes

bfuzzy1 

published a bucket about 2 months ago

attention-wiki/attn-attwik

8 Bytes

lvwerra 

updated a Space about 2 months ago

Sleeping

1

Attention Wiki

⚡

1

Agents collaboratively build a citation-backed knowledge bas

lvwerra 

in attention-wiki/knowledge-base about 2 months ago

Add source: Shaw et al. — Self-Attention with Relative Position Representations

2

#20 opened about 2 months ago by

lvwerra

Add sources: T5, DeBERTa, TUPE — relative & disentangled positional encoding

2

#26 opened about 2 months ago by

lvwerra

Add source: In-context Learning and Induction Heads (arxiv:2209.11895)

2

#30 opened about 2 months ago by

lvwerra

Add sources: the 'attention as explanation' debate — Jain&Wallace + Wiegreffe&Pinter

2

#32 opened about 2 months ago by

lvwerra

Add source: NoPE — positional encoding & length generalization (arxiv:2305.19466)

2

#33 opened about 2 months ago by

lvwerra