Multilingual-Multimodal-NLP/IndustrialCoder

InCoder-32B: Code Foundation Model for Industrial Scenarios

Model Summary

InCoder-32B (Industrial-Coder-32B) is the first 32B-parameter code foundation model purpose-built for industrial code intelligence. While general-purpose code LLMs excel at mainstream software tasks, they struggle with the unique demands of industrial programming — hardware semantics, specialized language constructs, strict resource constraints, and domain-specific correctness verification. InCoder-32B unifies code intelligence across five industrial domains:

Domain	Languages & Frameworks
🔧 Chip Design	Verilog, SystemVerilog, RTL
⚡ GPU Kernel Optimization	CUDA, Triton
🖥️ Embedded Systems	C/C++, ARM Cortex-M4, STM32
🔨 Compiler Optimization	x86-64 ASM, C/C++, LLVM-IR
📐 3D Modeling / CAD	CadQuery, OpenCascade, Python

InCoder-32B achieves competitive general-purpose performance while establishing the strongest open-source baselines across all evaluated industrial domains.

---

Key Results

General Code Benchmarks

Benchmark	InCoder-32B
SWE-bench Verified	74.8%
LiveCodeBench (Pass@1)	49.14%
BFCL v3	60.99%
HumanEval+	89.6%
MBPP+	78.3%
BigCodeBench (Full)	49.8%
τ²-bench (Retail)	85.1%
τ²-bench (Telecom)	86.8%

Industrial Code Benchmarks

Benchmark	Domain	InCoder-32B	Best Competing Open-Weight
VeriScope Score	Chip Design	80.7	83.2 (GLM-5)
VeriRepair Fix	Chip Design	80.0%	90.0% (GLM-5)
RealBench Syn@1 (Module)	Chip Design	74.8%	50.1% (Kimi-K2-Instruct)
ArchXBench (n)	Chip Design	51.0	50.0 (Claude-Sonnet-4.6)
CAD-Coder Compile	3D Modeling	82.0%	48.0% (Kimi-K2-Thinking)
CAD-Coder IoU	3D Modeling	53.5	20.0 (Kimi-K2-Thinking)
EmbedCGen Main	Code Optimization	35.2%	90.2% (GLM-5)
KernelBench L1	GPU Optimization	22.2%	16.2% (GLM-5)
KernelBench L2	GPU Optimization	36.0%	28.0% (KernelBench L2)
KernelBench L3	GPU Optimization	14.0%	8.0% (MiniMax-M2.5)
TritonBench G-call	GPU Optimization	18.5%	28.8% (Claude-Sonnet-4.6)

InCoder-32B leads all open-weight baselines on CAD-Coder and KernelBench (all three levels), and even surpasses the proprietary Claude-Sonnet-4.6 on CAD-Coder IoU and KernelBench L1/L2/L3.

---

Model Architecture

InCoder-32B adopts a standard decoder-only Transformer architecture with the following configuration:

Hyperparameter	Value
Parameters	~32B
Layers	64
Hidden Size	5,120
Intermediate Size	27,648
Attention Heads	40
KV Heads (GQA)	8
Head Dimension	128
Vocabulary Size	76,800
Max Context Length	131,072 (128K)
Activation	SiLU
Positional Encoding	RoPE (θ = 500,000)
Precision	BFloat16
Tie Embeddings	No

---

Training Pipeline

InCoder-32B is trained through a three-stage Code-Flow pipeline:

Stage 1 — Pre-training & Annealing

Industrial code corpora (Verilog, CUDA, firmware C, CadQuery scripts) are severely underrepresented in existing datasets like The Stack v2. We construct a dedicated data pipeline using:

• Three-step domain recall: rule-based filtering (file extensions, keywords like endmodule, __global__), FastText classifier, and semantic encoder retrieval
• OCR extraction from technical books and hardware reference manuals
• Multi-level deduplication: exact hash, MinHash LSH, repository-level fork consolidation, cross-source dedup
• Domain-specific validation: AST comparison, re-compilation, synthesis checks
• Data refinement: normalized formatting, cross-file dependency resolution, code-text alignment annotations

Training details:

• Hardware: 4,096 GPUs
• Objectives: Autoregressive LM + Fill-in-the-Middle (FIM)
• Learning rate: 3 × 10⁻⁴ (constant)
• Batch size: 2,048 globally
• Total tokens: 15T
• Curriculum: function-level → file-level → multi-file/project-level

Stage 2 — Mid-Training (Context Extension)

Context window is extended progressively from 8K to 128K tokens across two sub-stages, combined with domain-specific data synthesis:

Stage 2.1 — 8K → 32K:

• Targets file-level tasks: completing RTL modules, infilling kernel functions, generating testbenches
• Data mix: reasoning QA (40%), agent trajectories (20%), commits (15%), industrial artifacts (15%), FIM (10%)

Stage 2.2 — 32K → 128K:

• Unlocks long-context capabilities: extended debugging sessions, cross-module projects
• Graduated warm-up: long sequences start at 10%, linearly increases to 50%
• Data mix shifts toward long-context: agent trajectories (30%), FIM (25%), reasoning QA (25%)

Synthetic Industrial QA Pipeline:

1. Scenario specification — identified with practicing hardware engineers
2. Seed code generation — realistic RTL patterns, CUDA memory access idioms, interrupt-driven firmware
3. QA synthesis with automated verification — code execution validation, static analysis, logical consistency checks

Stage 3 — Post-Training

2.5M supervised fine-tuning samples are constructed directly from real industrial coding tasks with execution-grounded verification across four environments:

Environment	Toolchain
Chip Design	Icarus Verilog, Verilator, Yosys
GPU Optimization	NVIDIA A100, nvcc, Triton compiler
3D Modeling	CadQuery, OpenCascade
Embedded Systems	arm-none-eabi-gcc, Renode simulator (STM32F407)

SFT samples are organized into three categories:

• Direct solution — requirement-to-implementation
• Defect repair — failure-feedback-fix loop with closed-loop repair trajectories
• Performance & structural optimization — improving correct code for efficiency, readability, or architecture

---

Benchmarks

Industrial Benchmarks (New)

This release introduces several new industrial benchmarks:

• VeriScope — 568 Verilog generation problems across 5 difficulty levels (combinational logic → dual-core out-of-order RISC-V SoC with cache coherence). Graded via RTL simulation.
• VeriRepair — ~22K training / 300 test Verilog bug-fix samples with 4 major error categories and 20 error types.
• EmbedCGen — 500 bare-metal embedded C generation problems for STM32F407 (ARM Cortex-M4), evaluated via cross-compilation + Renode simulation.

General Benchmarks Evaluated

Category	Benchmarks
Code Generation	EvalPlus (HumanEval, MBPP), BigCodeBench, FullStackBench
Code Reasoning	CRUXEval, LiveCodeBench
Code Efficiency	Mercury
Text-to-SQL	Spider, BIRD
Agentic Coding	Terminal-Bench v1/v2, SWE-bench Verified
Tool Use	Mind2Web, BFCL v3, τ²-bench
Industrial	VeriScope, VeriRepair, RealBench, ArchXBench, CAD-Coder, EmbedCGen, SuperCoder, TritonBench, KernelBench

---

Usage

Installation

pip install transformers accelerate

Basic Inference

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Multilingual-Multimodal-NLP/IndustrialCoder"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

prompt = """Write a synthesizable Verilog module for a UART transmitter (8N1 protocol).
The module should accept 8-bit parallel data and serialize it onto a TX line."""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=1024,
    temperature=0.2,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Fill-in-the-Middle (FIM)

InCoder-32B supports FIM completion for code infilling tasks:

prefix = """// CUDA kernel for RMS Normalization
__global__ void rms_norm_kernel(float* output, const float* input, 
                                 const float* weight, int N, float eps) {
    int idx = blockIdx.x;
"""
suffix = """
    output[idx * N + tid] = normalized * weight[tid];
}"""

fim_prompt = f"<fim_prefix>{prefix}<fim_suffix>{suffix}<fim_middle>"
inputs = tokenizer(fim_prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Chat / Instruction Format

messages = [
    {
        "role": "user",
        "content": "Optimize this CUDA matrix multiplication kernel for an NVIDIA A100 using shared memory tiling with TILE_SIZE=32."
    }
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.1, do_sample=True)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

Limitations & Known Failure Modes

Based on analysis of 1,882 failure cases across 9 industrial benchmarks:

• Compilation & syntax errors: Dominant in Verilog tasks — 71% of RealBench failures involve malformed literals, incorrect port declarations, or bit-width mismatches.
• Incomplete API knowledge: 47% of EmbedCGen failures are linker errors from undefined or incorrectly typed HAL/CMSIS functions; 33% of TritonBench failures are NameErrors from incorrect Triton API usage.
• Format compliance: 46% of VeriScope failures are unparseable structured outputs where the required format is ignored entirely.
• Functional correctness under precise semantics: 79% of VeriRepair failures produce compilable but functionally incorrect code; most CAD-Coder failures stem from systematic Euler angle convention misinterpretation.
• Optimization gap: 33% of KernelBench failures produce functionally correct but insufficiently fast GPU kernels; 83% of SuperCoder failures result in the model copying input assembly without modification.

---

Ablation Findings

• Repository transition data outperforms static snapshots for planning tasks
• Mid-training reasoning trajectories improve robustness under distribution shift
• Thinking paths unlock emergent capabilities absent in standard instruction tuning
• Scaling industrial SFT data is a reliable performance driver across all 9 industrial benchmarks (83M → 167M → 250M tokens shows consistent improvement)

---

Citation

@article{yang2026incoder,
  title={InCoder-32B: Code Foundation Model for Industrial Scenarios},
  author={Yang, Jian and Zhang, Wei and Wu, Jiajun and Cheng, Junhang and Guo, Shawn 
          and Wang, Haowen and Gu, Weicheng and Du, Yaxin and Li, Joseph and Xu, Fanglin 
          and others},
  journal={arXiv preprint arXiv:2603.16790},
  year={2026}
}

---

Model Card Authors

Jian Yang, Wei Zhang, Jiajun Wu, Junhang Cheng, Shawn Guo, Haowen Wang, Weicheng Gu, Yaxin Du, Joseph Li, Fanglin Xu, Yizhi Li, Lin Jing, Yuanbo Wang, Yuhan Gao, Ruihao Gong, Chuan Hao, Ran Tao, Aishan Liu, Tuney Zheng, Ganqu Cui, Zhoujun Li, Mingjie Tang, Chenghua Lin, Wayne Xin Zhao, Xianglong Liu, Ming Zhou, Bryan Dai, Weifeng Lv

Affiliations: Beihang University, IQuest Research, Shanghai Jiao Tong University, ELLIS, University of Manchester, Shanghai Artificial Intelligence Laboratory, Sichuan University, Gaoling School of Artificial Intelligence (Renmin University of China), Langboat

---

License

This model is released under the Apache 2.0 License.