evaluerBench

What it is

An automated, local-first LLM evaluation and benchmarking pipeline powered by LangGraph and LangChain.

evaluerBench is a benchmarking tool using a state machine architecture to orchestrate multi-step evaluations, handle hardware-constrained cyclic execution, and route test cases through intelligent grading pathways. It exports structured JSON and raw HTML artifacts, designed to be consumed by component-driven static sites for developer portfolios and performance dashboards.

How it works

evaluerBench is built on a LangGraph StateGraph, which allows for complex routing, cyclic execution loops, and centralized state management.

• loadEvals Node: Ingests YAML test definitions, extracting prompts, configuration overrides, and display preferences.
• Dynamic Router: Determines whether to run sequentially (protecting low-throughput local hardware from OOM crashes) or fan-out via LangGraph's Send API for parallel Map-Reduce execution.
• generate Node: Streams responses from the local target model via LangChain's OpenAI-compatible driver. It calculates Time to First Token (TTFT) and active Tokens Per Second (TPS) independently of total latency.
• judge Node (Fast-Path Routing):
  • Zero-Cost Path: If an eval requires exact-match or regex validation, the node grades it instantly using zero tokens.
  • LLM-as-a-Judge Path: Evaluates open-ended reasoning or coding tasks using a secondary model enforcing strict structured JSON output via Zod.
• export Node: Aggregates the graph state and writes a cleanly structured bench.json into a timestamped run folder, extracting any visual demos (display_type: "html-iframe") into standalone .html artifacts.