Instructions to use AesSedai/GLM-5.1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AesSedai/GLM-5.1-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="AesSedai/GLM-5.1-GGUF",
	filename="IQ3_S/GLM-5.1-IQ3_S-00001-of-00007.gguf",
)

llm.create_chat_completion(
	messages = "No input example has been defined for this model task."
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use AesSedai/GLM-5.1-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf AesSedai/GLM-5.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf AesSedai/GLM-5.1-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf AesSedai/GLM-5.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf AesSedai/GLM-5.1-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf AesSedai/GLM-5.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf AesSedai/GLM-5.1-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf AesSedai/GLM-5.1-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf AesSedai/GLM-5.1-GGUF:Q4_K_M

Use Docker

docker model run hf.co/AesSedai/GLM-5.1-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use AesSedai/GLM-5.1-GGUF with Ollama:
```
ollama run hf.co/AesSedai/GLM-5.1-GGUF:Q4_K_M
```

Unsloth Studio

How to use AesSedai/GLM-5.1-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for AesSedai/GLM-5.1-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for AesSedai/GLM-5.1-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for AesSedai/GLM-5.1-GGUF to start chatting

How to use AesSedai/GLM-5.1-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf AesSedai/GLM-5.1-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "AesSedai/GLM-5.1-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use AesSedai/GLM-5.1-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf AesSedai/GLM-5.1-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default AesSedai/GLM-5.1-GGUF:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use AesSedai/GLM-5.1-GGUF with Docker Model Runner:
```
docker model run hf.co/AesSedai/GLM-5.1-GGUF:Q4_K_M
```

Lemonade

How to use AesSedai/GLM-5.1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AesSedai/GLM-5.1-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.GLM-5.1-GGUF-Q4_K_M

List all available models

lemonade list

GLM-5.1-GGUF / kld_data /llm_quantization_data.csv

AesSedai

Upload folder using huggingface_hub

0da9129 verified about 1 month ago

raw

history blame contribute delete

3.2 kB

	model_name,file_size_gb,bpw,Mean KLD_mean,0.1% KLD,0.1% Δp,1.0% KLD,1.0% Δp,10.0% KLD,10.0% Δp,25.0% Δp,5.0% KLD,5.0% Δp,75.0% Δp,90.0% KLD,90.0% Δp,95.0% KLD,95.0% Δp,99.0% KLD,99.0% Δp,99.9% KLD,99.9% Δp,"Cor(ln(PPL(Q)), ln(PPL(base)))",Maximum KLD,Maximum Δp,Mean KLD_std,Mean PPL(Q)-PPL(base)_mean,Mean PPL(Q)-PPL(base)_std,Mean PPL(Q)/PPL(base)_mean,Mean PPL(Q)/PPL(base)_std,Mean PPL(Q)_mean,Mean PPL(Q)_std,Mean PPL(base)_mean,Mean PPL(base)_std,Mean ln(PPL(Q)/PPL(base))_mean,Mean ln(PPL(Q)/PPL(base))_std,Mean Δp_mean,Mean Δp_std,Median KLD,Median Δp,Minimum KLD,Minimum Δp,RMS Δp_mean,RMS Δp_std,Same top p_mean,Same top p_std,common_fit_params,common_memory_breakdown_print,common_params_fit_impl,file_path,file_size_gib,ggml_cuda_init,kl_divergence,llama_context,llama_kv_cache,llama_model_loader,llama_perf_context_print,load,load_tensors,print_info,sched_reserve,system_info
	GLM-5.1-IQ3_S (aes_sedai),279.05476263936,2.96,0.262398,-1e-06,-98.467,2e-06,-87.099,8.6e-05,-25.857,-4.96,2.2e-05,-48.946,0.012,0.709111,3.989,1.351315,11.901,3.277228,40.638,6.219171,78.434,89.57,12.446607,97.43,0.001686,0.559204,0.00857,1.205353,0.003095,3.282336,0.018782,2.723132,0.0149,0.186773,0.002568,-5.699,0.052,0.034029,-0.127,-8e-06,-99.957,20.566,0.085,84.867,0.094,1.45,-11937440448.0,300132.676815,kld/GLM-5.1/wiki-test-raw/aes_sedai/GLM-5.1-IQ3_S.md,259.89,7680750.0,565512819216.0,9.45,512.0,-2150.0,34.0,0.9811,631897.7,1024.0,103.884,4.848561200112841e+50
	GLM-5.1-IQ4_XS (aes_sedai),361.43223537664005,3.84,0.099818,-2e-06,-86.77,0.0,-56.896,2.5e-05,-10.711,-1.559,6e-06,-22.537,0.05,0.248516,3.375,0.484126,9.332,1.391727,30.619,3.231633,66.862,95.94,8.413041,97.589,0.000754,0.169615,0.004528,1.062287,0.001665,2.892748,0.015981,2.723132,0.0149,0.060424,0.001568,-2.162,0.033,0.009711,-0.017,-8e-06,-99.825,12.667,0.07,90.958,0.076,0.84,-14129640448.0,377991.676815,kld/GLM-5.1/wiki-test-raw/aes_sedai/GLM-5.1-IQ4_XS.md,336.61,7680750.0,565512819216.0,9.45,512.0,-476.0,34.0,0.9811,641401.39,1024.0,118.394,4.848561200112841e+50
	GLM-5.1-Q4_K_M (aes_sedai),464.71546142720007,4.93,0.037406,-3e-06,-58.706,-0.0,-28.886,8e-06,-4.35,-0.373,2e-06,-9.827,0.148,0.091039,3.151,0.175733,7.839,0.523278,24.486,1.349837,52.058,98.46,5.044281,96.679,0.000308,0.031461,0.002644,1.011553,0.000973,2.754593,0.015142,2.723132,0.0149,0.011487,0.000962,-0.348,0.02,0.003027,-0.001,-2e-05,-97.94,7.59,0.051,94.489,0.06,0.83,-14129640448.0,475191.676815,kld/GLM-5.1/wiki-test-raw/aes_sedai/GLM-5.1-Q4_K_M.md,432.8,7680750.0,565512819216.0,9.45,512.0,-576.0,34.0,0.9811,653065.39,1024.0,112.754,4.848561200112841e+50
	GLM-5.1-Q5_K_M (aes_sedai),558.4316478259201,5.93,0.020247,-4e-06,-42.273,-1e-06,-19.476,5e-06,-2.863,-0.209,1e-06,-6.544,0.143,0.048455,2.536,0.093674,5.986,0.276042,18.434,0.808624,42.126,99.14,2.927605,85.486,0.000173,0.009288,0.001962,1.003411,0.000721,2.73242,0.015015,2.723132,0.0149,0.003405,0.000719,-0.09,0.015,0.001613,-0.0,-1.8e-05,-92.013,5.524,0.041,95.885,0.052,0.82,-14129640448.0,563391.676815,kld/GLM-5.1/wiki-test-raw/aes_sedai/GLM-5.1-Q5_K_M.md,520.08,7680750.0,565512819216.0,9.45,512.0,-676.0,34.0,0.9811,663649.39,1024.0,108.894,4.848561200112841e+50