| --- |
| license: mit |
| base_model: |
| - stabilityai/stable-diffusion-xl-base-1.0 |
| pipeline_tag: text-to-image |
| --- |
| # SarcasmDiffusion — SDXL Fused Meme Generator |
|
|
| **Model type:** Stable Diffusion XL (Base 1.0) fine‑tuned via **LoRA** (merged/fused) to learn the *visual* style of sarcastic/ironic memes. |
| **Author:** Ricardo Urdaneta (github.com/Ricardouchub) |
|
|
|
|
| --- |
|
|
| ## Overview |
|
|
| SarcasmDiffusion is a diffusion-based generative model focused on producing **clean meme-style photographs** that are suitable for **caption overlays** (text is added *after* generation). The model was LoRA‑fine‑tuned on a filtered and enriched subset of the *Hateful Memes* dataset to capture stylistic cues of humorous/ironic memes while **avoiding offensive content**. |
|
|
| - **Base:** `stabilityai/stable-diffusion-xl-base-1.0` |
| - **Fine‑tuning:** LoRA on the **UNet** only; **VAE** and **text encoders** are frozen. |
| - **Exported artifact:** **Fused SDXL** (no external LoRA required at inference). |
|
|
| > This model focuses on **style transfer for meme aesthetics** (composition, lighting, “stock-photo vibe”), *not* on rendering text inside images. Add titles/subtitles with your own overlay function or editor. |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| - Generating **meme-ready images** with space at the top/bottom for captions. |
| - Creative exploration of humorous/ironic visual setups controlled by prompts. |
| - Educational/portfolio use for **LoRA fine‑tuning workflows** with SDXL. |
|
|
| ### Out of Scope / Limitations |
| - **No text rendering inside the image** (explicitly discouraged via negative prompts). |
| - May produce **stock-like** aesthetics by design. |
| - Not suitable for generating or amplifying **harmful, hateful, or NSFW** content. |
| - As with all text-to-image systems, prompts with ambiguous semantics can yield unpredictable outputs. |
|
|
| --- |
|
|
| ## Training Summary |
|
|
| - **Base model:** SDXL Base 1.0 |
| - **LoRA rank / alpha / dropout:** `r=8`, `alpha=16`, `dropout=0.05` |
| - **Resolution:** 1024 (training); common inference at 768–896 for speed |
| - **Batch:** 1 (gradient accumulation = 4) |
| - **Steps:** ~9k (≈2 epoch on ~5k images) |
| - **Learning Rate:** 0.0001 |
| - **Precision:** fp16 (LoRA params kept in fp32 during training) |
| - **Optimizer:** AdamW |
| - **Scheduler:** cosine with warmup (recommended) |
| - **Frozen:** VAE, text_encoder, text_encoder_2 |
| |
| ### Data |
| - Source: *Hateful Memes* (Facebook AI). |
| - We **excluded** labeled hateful samples and applied **NLP enrichment**: |
| - Emotion scoring (GoEmotions distilled) and irony scoring (RoBERTa‑irony). |
| - Heuristics + percentiles → tones: `humor / irony / neutral`. |
| - Final training CSV: prompts balanced by tone; **negative prompts** to avoid text overlays, low‑quality artifacts, watermarks/logos, and unsafe content. |
| |
| > The dataset is **not** included here. Please obtain *Hateful Memes* under its original terms and reproduce the preprocessing if needed. |
| |
| --- |
| |
| ## Safety, Ethics & Mitigations |
| |
| - Hateful labels were filtered out **negative prompts** is used to avoid NSFW/hate/text overlays. |
| - Despite mitigations, **misuse is possible**. Users are responsible for **prompting responsibly** and complying with local laws and platform policies. |
| - Do not use the model to create defamatory, harassing, discriminatory, or otherwise harmful imagery. |
| |
| **Known risks:** dataset biases may remain; aesthetic biases (stock-photo look); occasional failure to respect negative prompts. |
| |
| --- |
| |
| ## How to Use |
| |
| ```python |
| from diffusers import AutoPipelineForText2Image |
| import torch |
| |
| pipe = AutoPipelineForText2Image.from_pretrained( |
| "Ricardouchub/SarcasmDiffusion", |
| torch_dtype=torch.float16 |
| ).to("cuda") # use "cpu" if no GPU |
| |
| prompt = ( |
| "sarcastic meme about checking the fridge for the third time, " |
| "centered subject, plain background, high-contrast photo, stock photo style" |
| ) |
| negative = "nsfw, hate speech, slur, watermark, logo, low quality, blurry, busy background, text overlay" |
| |
| g = torch.Generator(device=pipe.device).manual_seed(123) |
| image = pipe(prompt, |
| negative_prompt=negative, |
| num_inference_steps=22, |
| guidance_scale=6.3, |
| width=896, height=896, |
| generator=g).images[0] |
| |
| image.save("sample.png") |
| ``` |
| |
| ### Prompting Tips |
| - Add **layout hints**: “centered subject”, “plain background”, “space at top and bottom”. |
| - Keep **negative prompts** to avoid logos/text/NSFW. |
| - Use seeds for reproducibility; `steps=18–28`, `guidance=5.5–7.5`, `size=768–1024`. |
| |
| --- |
| |
| ## Environment & Compatibility |
| |
| To ensure full compatibility when loading this model (fused SDXL with LoRA merged), use the following library versions: |
| |
| | Library | Recommended Version | Notes | |
| |----------|--------------------|-------| |
| | **Python** | 3.10 – 3.12 | Tested on Colab (Python 3.12) | |
| | **PyTorch** | 2.6.0 + CUDA 12.4 | Any CUDA ≥ 12 works | |
| | **diffusers** | **0.35.1** | Core inference & model loading | |
| | **transformers** | **4.45.2** | Required for SDXL CLIPTextEncoder compatibility | |
| | **accelerate** | **1.10.1** | Device and fp16 inference management | |
| | **huggingface_hub** | **0.23.5** | Compatible with diffusers 0.35.x | |
| | **safetensors** | ≥ 0.4.5 | For secure model weights loading | |
| |
| **Install in Colab or local environment:** |
| |
| ```bash |
| pip install "diffusers==0.35.1" "transformers==4.45.2" "accelerate==1.10.1" "huggingface_hub==0.23.5" safetensors |
| ``` |
| |
| > **Important:** |
| > Using newer versions (e.g., `transformers ≥ 4.56`) may break compatibility due to API changes in `CLIPTextModel` (`offload_state_dict` argument). |
| > Always match the versions above for smooth loading. |
| |
| --- |
| |
| ## License |
| |
| - **Code:** MIT |
| - **Model weights:** follow the base model’s license (Stability AI / SDXL Base 1.0). |
| - **Data:** Users must obtain *Hateful Memes* from its source and agree to its terms. |
| |
| > By using this model, you agree not to generate content that is illegal, harmful, or violates rights of others. |
| |
| --- |
| |
| ## Evaluation |
| |
| Qualitative assessment via fixed prompt sheets (humor/irony/neutral). Suggested automatic metrics for future work: CLIP‑score vs. caption, aesthetic predictors, and human preference studies. |
| |
| --- |
| |
| ## Acknowledgments |
| |
| - Stability AI — SDXL Base 1.0 |
| - Hugging Face — Diffusers, Accelerate, PEFT |
| - Facebook AI — Hateful Memes dataset |
