rtx6kpro
A community wiki and toolset helping developers run large models like Qwen3.5-397B on multiple NVIDIA RTX PRO 6000 GPUs without NVLink, covering optimizations, quantizations, and benchmarks.
Tool overview
What it does: rtx6kpro is a community-driven wiki and toolset that documents how to run massive language models—such as Qwen3.5-397B, Kimi-K2.5, GLM-5, and DeepSeek v4 Flash—on NVIDIA RTX PRO 6000 multi-GPU setups without NVLink. It includes ready-to-use model configurations, sm_120 kernel patches, NVFP4 quantization recipes, and tuning instructions for inference engines like SGLang and vLLM.
Strengths: It enables ultra-large model inference on PCIe-only topologies, avoiding the need for costly NVLink bridges. Community-contributed high-throughput kernels and model-optimized serving stacks can deliver decode speeds over 150 tokens/s and prefill at 10,000+ tokens/s on 8× RTX PRO 6000. The project’s Discord community (over 1,300 members) offers real-time troubleshooting.
Weaknesses/Risks: The main barrier is hardware cost—an 8× RTX PRO 6000 system is expensive, and most guides assume you already own or rent such hardware (e.g., via Verda). As a community project, there is no official support, and certain optimizations (e.g., NVFP4 MLA workarounds) can be fragile and require manual tuning.