ankur1423/fine-tune-test

Solar FAQ — Llama-3.1-8B LoRA Fine-tune

A Llama-3.1-8B-Instruct model fine-tuned with LoRA on a solar energy FAQ dataset using MLX-LM on Apple Silicon.

Base model	meta-llama/Meta-Llama-3.1-8B-Instruct
Format	float16 safetensors (safe — no pickle)
Size	~15 GB (float16)
Fine-tune method	LoRA rank 8, 8 layers
Domain	Solar energy FAQ
Languages	English

Smaller version available: GGUF Q4_K_M (4.6 GB) — runs on CPU, Mac, Windows, Linux without GPU.

---

Model Overview

This model is a LoRA fine-tune experiment on top of Meta's Llama-3.1-8B-Instruct, trained on a small domain-specific solar energy FAQ dataset (~62 Q&A pairs). It answers questions about solar products, manufacturing processes, and company operations.

Outside the training domain it falls back to standard Llama-3.1 behaviour.

What it can do

• Answer solar energy FAQ questions accurately
• Explain solar manufacturing concepts (BOM, PPC, audits, etc.)
• Provide concise, professional responses to domain-specific queries
• Multi-turn conversation with context retention

What it cannot do

• General-purpose assistant (use base Llama-3.1 for that)
• Image / audio / video understanding
• Real-time or internet-connected queries

---

Getting Started

Installation

# GPU (NVIDIA) or CPU:
pip install transformers torch accelerate bitsandbytes

# Apple Silicon (recommended — faster with MLX):
pip install mlx-lm

Quick Inference (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ankur1423/fine-tune-test"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",        # auto: GPU if available, else CPU
)

def ask(question: str) -> str:
    messages = [
        {"role": "system",
          "content": "You are a knowledgeable assistant for a solar energy company. "
                     "Answer questions accurately about solar products, manufacturing, and company operations."},
        {"role": "user", "content": question},
    ]
    prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        output = model.generate(
            **inputs,
            max_new_tokens=512,
            temperature=0.1,
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    new_tokens = output[0][inputs["input_ids"].shape[1]:]
    return tokenizer.decode(new_tokens, skip_special_tokens=True).strip()

print(ask("What is a BOM?"))
print(ask("What is PPC in solar manufacturing?"))
print(ask("Why are internal audits important?"))

4-bit Quantized Inference (saves ~12 GB RAM)

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "ankur1423/fine-tune-test"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

# Same ask() function as above — uses ~5 GB VRAM instead of 15 GB

Apple Silicon — MLX (fastest on Mac)

from mlx_lm import load, generate
from mlx_lm.generate import make_sampler

model, tokenizer = load("ankur1423/fine-tune-test")

SYSTEM = "You are a knowledgeable assistant for a solar energy company."

def ask(question: str) -> str:
    prompt = (
        "<|begin_of_text|>"
        "<|start_header_id|>system<|end_header_id|>\n\n"
        + SYSTEM + "<|eot_id|>"
        "<|start_header_id|>user<|end_header_id|>\n\n"
        + question + "<|eot_id|>"
        "<|start_header_id|>assistant<|end_header_id|>\n\n"
    )
    return generate(
        model, tokenizer,
        prompt=prompt,
        max_tokens=512,
        sampler=make_sampler(temp=0.1, top_p=0.9),
    )

print(ask("What is a BOM?"))

Multi-turn Chat (transformers)

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

model_id = "ankur1423/fine-tune-test"
SYSTEM = "You are a knowledgeable assistant for a solar energy company."

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.float16,
        bnb_4bit_quant_type="nf4",
    ),
    device_map="auto",
)

history = [{"role": "system", "content": SYSTEM}]

while True:
    user = input("You: ").strip()
    if not user or user.lower() in {"exit", "quit"}:
        break
    history.append({"role": "user", "content": user})

    prompt = tokenizer.apply_chat_template(
        history, tokenize=False, add_generation_prompt=True
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs, max_new_tokens=512,
            temperature=0.1, top_p=0.9, do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    response = tokenizer.decode(
        out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
    ).strip()
    print(f"Assistant: {response}\n")
    history.append({"role": "assistant", "content": response})

---

Platform Support

Platform	Method	RAM / VRAM	Speed
Mac M1/M2/M3/M4	MLX (4-bit)	5 GB	Fast
NVIDIA GPU (Linux/Windows)	transformers 4-bit	5–6 GB VRAM	Fast
Google Colab T4	transformers 4-bit	~6 GB VRAM	Fast
Kaggle P100	transformers 4-bit	~6 GB VRAM	Fast
CPU — any OS	transformers float16	16 GB RAM	Slow
Any platform (recommended)	GGUF 4.6 GB	6 GB RAM	Fast/OK

Tip: For CPU or low-VRAM machines, use the GGUF version — same quality, 4.6 GB, no GPU needed.

---

Recommended Generation Parameters

Parameter	Value	Notes
temperature	0.1	Low → factual, consistent
top_p	0.9	Nucleus sampling
max_new_tokens	256–512	FAQ answers are concise
do_sample	True	Required when temperature > 0

Raise temperature to 0.5–0.7 for more varied / creative responses.

---

Prompt Format

This model uses the Llama-3 chat template with <|eot_id|> as the stop token.

tokenizer.apply_chat_template() handles formatting automatically.

Raw format:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a knowledgeable assistant for a solar energy company.<|eot_id|>
<|start_header_id|>user<|end_header_id|>

What is a BOM?<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>

Stop token: <|eot_id|>

---

Training Details

Fine-tuning Process

The model was fine-tuned using LoRA (Low-Rank Adaptation) — only a small set of adapter weights are trained; the base model weights are frozen. This allows high-quality fine-tuning with minimal compute and memory.

Base model	meta-llama/Meta-Llama-3.1-8B-Instruct
Fine-tuning method	LoRA
LoRA rank	8
LoRA layers	8 (attention layers)
Dataset size	62 train + 6 validation (68 total Q&A pairs)
Iterations	300
Learning rate	1e-4 (cosine decay → 1e-5)
Warmup steps	30
Batch size	2
Max sequence length	1024 tokens
Framework	MLX-LM
Training hardware	MacBook M4 16 GB unified memory
Training time	~20 minutes

Dataset

The training dataset consists of ~68 solar energy FAQ Q&A pairs covering topics such as:

• Bill of Materials (BOM) and procurement
• Production Planning & Control (PPC)
• Solar panel manufacturing processes
• Quality control and internal audits
• Company operations and workflows

Data format — Llama-3 chat template, one Q&A pair per record:

{"text": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\n[system]<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n[question]<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n[answer]<|eot_id|>"}

---

Ethics and Safety

• Model is domain-specific and not a general-purpose assistant
• Answers are based on training data — verify critical information independently
• Not intended for medical, legal, or financial advice
• Solar energy domain only — out-of-domain queries fall back to base Llama-3 behaviour
• Inherits all safety characteristics of the base meta-llama/Meta-Llama-3.1-8B-Instruct model

---

Usage and Limitations

Intended Use

• Solar energy company FAQ chatbot
• Internal knowledge base assistant
• Learning / research on domain-specific LoRA fine-tuning with MLX

Out-of-Scope Use

• General-purpose assistant (use base Llama-3.1 instead)
• Medical, legal, or financial advice
• Real-time data retrieval (model has no internet access)
• Languages other than English

Known Limitations

• Small dataset (~68 pairs) — may not generalize to all solar topics
• English only
• Float16 format requires ~15 GB disk and ~6 GB VRAM / 16 GB RAM
• Apple Silicon only for MLX inference (use transformers on other platforms)

---

License

This model is derived from Meta Llama 3.1, which is licensed under the Meta Llama 3 Community License. Use is subject to Meta's acceptable use policy.