Crucible

What it is

A distributed orchestration platform designed to automate the evaluation of Large Language Models (LLMs) and publish the results.

Crucible leverages the Model Context Protocol (MCP) to allow external agents to trigger evaluations. These requests are picked up by a Temporal orchestration engine which manages a complex, multi-step workflow. The system provisions models on local hardware (via a Lemonade server), executes benchmarking using evaluerBench, offloads large evaluation artifacts to an S3-compatible MinIO instance, and finally publishes the results atomically directly to a GitHub repository (such as this Astro site) for public viewing.

How it works

The entire infrastructure runs in Docker containers, utilizing a microservices architecture to handle state management and execution:

• Temporal Orchestration: A Temporal server (backed by PostgreSQL) manages the complex, multi-step orchestration of the evaluation pipeline, ensuring fault-tolerance and resumability.
• MinIO Artifact Storage: An S3-compatible MinIO instance stores large evaluation artifacts, seamlessly handling outputs generated by the evaluation worker.
• Evaluation Worker: A dedicated Temporal worker executes the core evaluation logic using evaluerBench against models provisioned via a local hardware server.
• FastMCP Gateway: Exposes the system's capabilities to external agents using the Model Context Protocol, enabling AI agents to autonomously trigger and monitor evaluation runs.