gg-tt (gg-tt)

posted an update 5 days ago

Post

743

DeepSeek-V4-Flash can now run 2× faster locally with DSpark! ⚡️

DSpark enables V4-Flash-0731 GGUFs to generate ~1.4–2× faster with no accuracy change.

DeepSeek-V4-Flash-0731 can reach at 120 tokens/s.

GGUFs: unsloth/DeepSeek-V4-Flash-0731-GGUF
Guide: https://unsloth.ai/docs/models/deepseek-v4

mlabonne

posted an update 6 days ago

Post

3042

Deploy local agents everywhere with LiquidAI/LFM2.5-2.6B

Unlike agents that depend on cloud APIs, local agents give you free inference, low latency, and real privacy.

Removing the per-token cost changes how developers build: agents can now be massively parallelized on local hardware, running background tasks that burn through millions of tokens at no marginal cost!

7 replies

·

danielhanchen

posted an update 11 days ago

Post

1494

We compared 1-bit Kimi K3 to Claude Opus 5 and GPT 5.6. 🤯

We gave 4 models the same prompt: Create a glass aquarium whose side panel develops a visible crack and then bursts...

1-bit Kimi K3 GGUF ran locally on 4x B200s at 36 tok/s.

GGUF: unsloth/Kimi-K3-GGUF
GitHub repo: https://github.com/unslothai/unsloth

1 reply

·

danielhanchen

posted an update 13 days ago

Post

4088

Kimi K3 can now be run locally! ✨

The 1-bit model retains ~78.9% accuracy after we shrunk it from 1.56TB to 594GB (-62% size).

Run on a Mac Studio connected with 128GB RAM device. Kimi K3 is the strongest open model to date.

GGUF: unsloth/Kimi-K3-GGUF
Guide: https://unsloth.ai/docs/models/kimi-k3

5 replies

·

danielhanchen

posted an update 22 days ago

Post

4954

Introducing Unsloth for AMD 🚀
You can now train & run LLMs on your AMD hardware

• We collaborated with AMD to enable you to train & run 500+ models on AMD GPUs
• Works on Windows, WSL, Linux
• Train Qwen, Gemma on just 3GB VRAM

GitHub: https://github.com/unslothai/unsloth
Blog + Guide: https://unsloth.ai/docs/basics/amd

3 replies

·

danielhanchen

posted an update 24 days ago

Post

5995

Gemma 4 is now faster and much more accurate! 🚀

Google made huge improvements to tool-calling and chat accuracy, reliability + speed.
To get fixes, re-download our updated GGUF, MLX, NVFP4 quants!

Unsloth quants: https://huggingface.co/collections/unsloth/gemma-4
Gemma 4 Guide: https://unsloth.ai/docs/models/gemma-4

8 replies

·

danielhanchen

posted an update 28 days ago

Post

4813

We’re releasing Gemma 4 NVFP4 quants that run 1.5× faster on your GPU.

Gemma-4-12B NVFP4 works on 11GB VRAM.
26B-A4B hits 13K tok/s (B200).

Unsloth NVFP4 enables faster, more accurate 4-bit Blackwell inference.

Blog: https://unsloth.ai/docs/basics/nvfp4
Gemma NVFP4: https://huggingface.co/collections/unsloth/nvfp4

3 replies

·

danielhanchen

posted an update about 1 month ago

Post

4362

We’re releasing new Qwen3.6 quants that run 2.5× faster on your GPU. ⚡

Qwen3.6-27B NVFP4 runs on 24GB VRAM.
35B-A3B can hit 17,561 tok/s (B200).

We also improved accuracy, tool calling, agent use, and looping.

Qwen3.6 NVFP4: https://huggingface.co/collections/unsloth/nvfp4
Guide: https://unsloth.ai/docs/models/qwen3.6#nvfp4

1 reply

·

osanseviero

authored a paper about 1 month ago

Gemma 4 Technical Report

Paper • 2607.02770 • Published Jul 2 • 76

osanseviero

submitted a paper to Daily Papers about 1 month ago

Gemma 4 Technical Report

Paper • 2607.02770 • Published Jul 2 • 76

danielhanchen

posted an update about 1 month ago

Post

6325

DeepSeek-V4 can now run locally with Unsloth GGUFs! 🐳

Run lossless DeepSeek-V4-Flash on 168GB RAM or
3-bit works on 110GB Mac, RAM, VRAM setups.

Run via Unsloth Studio or llama.cpp.

GGUF: unsloth/DeepSeek-V4-Flash-GGUF
Guide: https://unsloth.ai/docs/models/deepseek-v4

danielhanchen

posted an update about 2 months ago

Post

3391

1-bit GLM-5.2 GGUF vs. Claude 4.8 Opus vs. GPT-5.5

We gave 3 models the same prompt and compared one-shot outputs.

The 1-bit GLM-5.2 GGUF ran locally on a Mac Studio M3 Ultra with 256GB RAM at ~21.6 tok/s.

Which output do you like best?
GGUF: unsloth/GLM-5.2-GGUF

3 replies

·

danielhanchen

posted an update about 2 months ago

Post

4627

Google's new DiffusionGemma can now run at 2000+ tokens/sec! ⚡

We made local DiffusionGemma inference 1.8× faster.
Run it on 18GB RAM via Unsloth Studio.

GitHub: https://github.com/unslothai/unsloth
Guide: https://unsloth.ai/docs/models/diffusiongemma

4 replies

·

danielhanchen

posted an update 2 months ago

Post

1207

Google releases DiffusionGemma.✨
The new 26B-A4B diffusion text model runs locally on 18GB RAM.

Run with 4x faster text generation, thinking, image, video and 256K context. Run and train via Unsloth Studio.

GGUF: unsloth/diffusiongemma-26B-A4B-it-GGUF
Guide: https://unsloth.ai/docs/models/diffusiongemma

1 reply

·

danielhanchen

posted an update 2 months ago

Post

4313

Google releases Gemma 4 QAT. ✨
You can now run Gemma 4 at 3x less memory with near original performance.

QAT makes it possible to run Gemma 4 26B-A4B on 16GB RAM.

GGUFs: https://huggingface.co/collections/unsloth/gemma-4-qat
QAT Guide: https://unsloth.ai/docs/models/gemma-4/qat

1 reply

·

danielhanchen

posted an update 2 months ago

Post

9374

Gemma 4 12B can now run locally on just 8GB RAM via Dynamic GGUFs.

Google's new model, Gemma 4 12B Unified supports image, audio and 256K context.
You can run and train the model via Unsloth Studio.

GGUF: unsloth/gemma-4-12b-it-GGUF
Guide: https://unsloth.ai/docs/models/gemma-4

5 replies

·

srush

authored a paper 2 months ago

Composer 2 Technical Report

Paper • 2603.24477 • Published Mar 25 • 20

emozilla

authored a paper 3 months ago

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

Paper • 2604.27263 • Published May 14 • 11

danielhanchen

posted an update 3 months ago

Post

2849

Qwen3.6 MTP is here! Run locally on 20GB RAM. ⚡️

MTP enables Qwen3.6 to generate ~1.4–2.2× faster with no accuracy change.

Qwen3.6-27B: unsloth/Qwen3.6-27B-MTP-GGUF
Qwen3.6-35B-A3B: unsloth/Qwen3.6-35B-A3B-MTP-GGUF
Guide: https://unsloth.ai/docs/models/qwen3.6#mtp-guide

2 replies

·

tomaarsen

posted an update 3 months ago

Post

1560

🤗 Announcing the Ettin Reranker family: six new state-of-the-art CrossEncoder rerankers for search from 17M to 1B parameters, plus the full training data and the ~150-line recipe. Built on the Ettin ModernBERT encoders, Apache 2.0. Details:

All six were trained with the same single-stage pointwise MSE distillation recipe, with mixedbread-ai/mxbai-rerank-large-v2 (1.54B) as the teacher. Only the learning rate and per-device batch size change between sizes. The 1B student matches the teacher within 0.0001 NDCG@10 on MTEB(eng, v2) Retrieval, the 150M is the strongest reranker I tested in the under-600M range, and the 17M beats the 33M ms-marco-MiniLM-L12-v2 by +0.051 NDCG@10 at roughly half the parameter count.

Speed matters as much as quality for a reranker, since it determines whether the model fits the latency budget between retrieval and showing results. Our 17M is the fastest reranker in the whole comparison at 7517 pairs/sec on an H100. Our 150M runs 2.3x faster than the two other 150M ModernBERT-base rerankers (gte-reranker-modernbert-base and granite-embedding-reranker-english-r2) because the modular Transformer module propagates unpadded inputs through every layer rather than just the FA2 attention kernel. And our 1B is 2.4x faster than its 1.5B teacher while matching it on quality.

I bootstrapped the training recipe with the new train-sentence-transformers Agent Skill shipped in Sentence Transformers v5.5.0. Install it with hf skills add train-sentence-transformers --claude and ask Claude Code (or Codex / Cursor / Gemini CLI) to fine-tune a SentenceTransformer, CrossEncoder, or SparseEncoder model on your data.

I wrote a blog post walking through usage, results across six embedder pairings, the speed story, and the complete training script. Check it out, or just point your Agent to the URL:

https://huggingface.co/blog/ettin-reranker

Collection: https://huggingface.co/collections/cross-encoder/ettin-rerankers

gg-tt

AI & ML interests

Recent Activity

Gemma 4 Technical Report

Gemma 4 Technical Report

Composer 2 Technical Report

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation

AI & ML interests

Recent Activity

Team members 95

gg-tt's activity