KernelBench
A benchmark that evaluates LLMs' ability to write high-performance GPU kernels, helping AI researchers quantify correctness and speedup of automatically generated CUDA/Triton operators.
Tool overview
KernelBench is a dedicated benchmark for autonomous coding agents that assesses whether LLMs can produce efficient GPU kernels. It standardizes the evaluation with roughly 250 operator tasks of varying difficulty—covering element-wise operations, reductions, matrix multiplication etc.—and provides unified metrics such as pass@k and fast_p, enabling fair comparisons across models.
Its main advantage is the systematic evaluation framework, which has been adopted by NVIDIA, Meta, DeepSeek, Moore Threads and others to benchmark cutting-edge models. Researchers can quickly measure the real coding capability of their LLMs on low-level GPU programming and accelerate the automation of performance-critical kernels.
Limitations include a focus on standalone operator optimization that may not reflect the diversity of production kernels; only the generation phase is evaluated, not debugging, deployment or cross‑architecture porting. Over‑optimizing for KernelBench may lead to overfitting and poor generalization on real‑world tasks. Additionally, metric adjustments (e.g. raising the fast_p threshold) can affect the comparability of historical results.