Compose The Libraries: Run A Benchmark Study#

This tutorial runs the smallest deterministic handoff from peer Problems and Agents inputs through Experiments artifacts into Analysis. It is the recommended compatibility smoke test and the best starting point for a new composed study.

What You Will Learn#

  • Load a packaged design benchmark through the umbrella namespace.

  • Define a study around the public seeded baseline agent.

  • Execute deterministic replicates and export canonical artifacts.

  • Validate event rows and compute a condition metric from those artifacts.

  • Write a human-readable study summary next to the raw tables.

Install And Run#

Download canonical_artifact_flow.py, open its containing folder in VS Code, and use the integrated terminal:

python -m pip install design-research==0.4.0
python canonical_artifact_flow.py

Walkthrough#

The umbrella keeps the imports coherent while each component retains ownership of its behavior. Problems supplies the benchmark, Agents supplies the baseline participant, Experiments owns execution and export, and Analysis consumes the versioned event artifact.

 1"""Smallest deterministic peer-input artifact flow through the umbrella."""
 2
 3from __future__ import annotations
 4
 5from pathlib import Path
 6from statistics import mean
 7
 8import design_research as dr
 9
10PROBLEM_ID = "decision_laptop_design_profit_maximization"
11AGENT_ID = "SeededRandomBaselineAgent"
12STUDY_ID = "canonical_artifact_flow"
13OUTPUT_DIR = Path("artifacts") / "examples" / STUDY_ID
14PRIMARY_METRIC = "primary_outcome"
15
16
17def main() -> None:
18    """Run one packaged problem through the public umbrella package family."""
19    # Start with a packaged benchmark from design-research-problems. The umbrella
20    # import keeps the example focused on the workflow instead of package wiring.
21    problem = dr.problems.get_problem(PROBLEM_ID)
22
23    # Build the smallest useful study: one problem, one public agent, and two
24    # deterministic replicates so the analysis layer has real rows to consume.
25    study = dr.experiments.build_strategy_comparison_study(
26        dr.experiments.StrategyComparisonConfig(
27            study_id=STUDY_ID,
28            title="Canonical Artifact Flow",
29            description="Run the minimal composed ecosystem path and validate its artifacts.",
30            bundle=dr.experiments.BenchmarkBundle(
31                bundle_id="canonical-artifact-flow",
32                name="Canonical Artifact Flow",
33                description="One packaged benchmark and one public baseline agent.",
34                problem_ids=(PROBLEM_ID,),
35                agent_specs=(AGENT_ID,),
36            ),
37            run_budget=dr.experiments.RunBudget(replicates=2, parallelism=1, max_runs=2),
38            output_dir=OUTPUT_DIR,
39        )
40    )
41
42    # Materialize the abstract recipe into condition rows, then execute those
43    # rows through design-research-experiments.
44    conditions = dr.experiments.build_design(study)
45    results = dr.experiments.run_study(
46        study,
47        conditions=conditions,
48        checkpoint=False,
49        show_progress=False,
50    )
51
52    # Export canonical analysis tables. This is the main ecosystem handoff:
53    # experiments writes artifacts that analysis can read directly.
54    artifacts = dr.experiments.export_analysis_tables(
55        study,
56        conditions=conditions,
57        run_results=results,
58        output_dir=study.output_dir / "analysis",
59        validate_with_analysis_package=True,
60    )
61
62    # Validate the exported event table and build the metric summary directly
63    # from artifacts, without asking the user to load the CSV tables.
64    event_report = dr.analysis.validate_experiment_events(artifacts["events.csv"])
65    metric_rows = dr.analysis.build_condition_metric_table_from_artifacts(
66        artifacts["events.csv"],
67        metric=PRIMARY_METRIC,
68        condition_column="agent_id",
69    )
70
71    # Reporting helpers live with experiments because they know the study shape,
72    # while analysis owns the statistical and tabular transforms.
73    summary_path = dr.experiments.write_markdown_report(
74        study.output_dir,
75        "canonical_artifact_flow_summary.md",
76        dr.experiments.render_markdown_summary(study, results),
77    )
78
79    values = [float(row["value"]) for row in metric_rows]
80    successes = sum(result.status.value == "success" for result in results)
81
82    # Keep terminal output compact: enough to confirm the packages worked
83    # together and point readers at the generated artifacts.
84    print("Canonical artifact flow:", study.study_id)
85    print("Runtime flow: problems + agents -> experiment artifacts -> analysis")
86    print("Problem:", problem.metadata.title)
87    print("Agent:", AGENT_ID)
88    print("Runs:", len(results), f"({successes} success)")
89    print(f"Mean {PRIMARY_METRIC}:", f"{mean(values):.4f}")
90    print("Event rows valid:", event_report.is_valid, f"(rows={event_report.n_rows})")
91    print("Summary report:", summary_path.name)
92    print("Artifacts directory:", artifacts["events.csv"].parent)
93
94
95if __name__ == "__main__":
96    main()

Selected Output#

The script also prints the selected problem and agent, mean outcome, and output directory. These lines capture the main completion checks:

Canonical artifact flow: canonical_artifact_flow
Runtime flow: problems + agents -> experiment artifacts -> analysis
Runs: 2 (2 success)
Event rows valid: True (rows=2)
Summary report: canonical_artifact_flow_summary.md

Inspect artifacts/examples/canonical_artifact_flow/analysis after the run. Those files are the durable boundary between study execution and downstream analysis.

Next, keep the same artifact-first boundary while comparing longer process traces in Compose The Libraries: Compare Design Processes.