Bench + Loop — Research Eval Framework

Problem

Research engines claim quality with no cheap, repeatable proof.

Approach

Bench: JSON cases (id + query + regex contracts + minimums + judge criteria), 1-method engine protocol, cross-product runs, markdown + diffable JSON. 10 cases: recency, synthesis, citation-density, post-cutoff, long-form, honest-deficit, conflict, multi-step code. Loop: Engine + Judge + Reformulator + Acceptance (threshold / time-budget). Zero-key stubs, Ollama adapter.

Code

From src/deep_research_bench/engine.py — the whole framework rests on a minimal protocol:

@dataclass(frozen=True)
class EngineAnswer:
    """An engine's response to a query."""

    text: str
    citations: List[str] = field(default_factory=list)
    metadata: Dict[str, Any] = field(default_factory=dict)
    # …any object with .text / .citations / .metadata
    # duck-types into the runner — engines wrap Perplexity,
    # OpenAI Deep Research, or your own private system

Preview

Live demo isn’t deployed — it’s a Python framework, so “preview” is a terminal run: install from GitHub (source-only release, Python 3.8+, zero required runtime deps), run the suite, get markdown + JSON reports across the 10 public cases. Zero-effort mock passes 0/10 — the scoring rejects weak answers rather than rubber-stamping. Companion Rust infra: evergreen-watchdog-rs, racefree-filelock-rs.

Results

Zero-effort mock passes 0/10 — scoring rejects weak answers rather than rubber-stamping. Iterative refinement beats single-shot on multi-part/citation-sensitive cases.

Links

• Bench: https://github.com/dnarsh/deep-research-bench
• Loop: https://github.com/dnarsh/refinement-loop
• Companion infra: evergreen-watchdog-rs, racefree-filelock-rs (Rust, crash-safe)