benchmark-qed

benchmark-qed automates RAG evaluation by running predefined benchmarks that measure retrieval accuracy and generation quality, standardizing testing and reducing manual work.

As an open-source project, it provides a free, adaptable framework for comparing different RAG configurations, vector stores, or LLM backends, potentially lowering evaluation costs.

Currently, the project appears early-stage with no visible community activity. Documentation and benchmark coverage may be limited, requiring users to invest effort in setup and debugging.

It suits developers and researchers accustomed to Python and command-line tools, but is not a turnkey solution for non-technical users; deployment demands computational resources.