Compose The Libraries: Compare Design Processes#

Outcome scores alone can hide how a design process changed. This tutorial runs two deterministic process treatments, exports their action traces, fits one Markov chain per condition, and compares the transition matrices.

What You Will Learn#

  • Use a packaged ideation problem as shared task context.

  • Bind deterministic participant callables to experiment conditions.

  • Export event sequences through the canonical artifact contract.

  • Fit and compare condition-specific Markov chains from artifacts.

  • Read process and outcome summaries from the same event table.

Install And Run#

Download long_agent_markov_comparison.py, open its containing folder in VS Code, and use the integrated terminal:

python -m pip install design-research==0.4.0
python long_agent_markov_comparison.py

Core Orchestration#

The complete runnable source includes the deterministic transition policies and helper functions. The excerpt below shows the study, execution, artifact, and analysis path.

 1def main() -> None:
 2    """Run two agent treatments, then compare their transition matrices."""
 3    # The problem package supplies the real design-task context. The agents below
 4    # stay scripted so the example can focus on process traces and analysis.
 5    problem = dr.problems.get_problem(PROBLEM_ID)
 6
 7    # Both treatments get the same action vocabulary and run budget. Only the
 8    # transition tendencies differ, which makes the Markov comparison meaningful.
 9    study = dr.experiments.Study(
10        study_id=STUDY_ID,
11        title="Long Agent Markov Comparison",
12        description=(
13            "Run long synthetic traces with the same action vocabulary and compare "
14            "condition-specific Markov-chain matrices from exported artifacts."
15        ),
16        factors=(
17            dr.experiments.Factor(
18                name="agent_id",
19                description="Agent process treatment.",
20                levels=(
21                    dr.experiments.Level(name=BASELINE_AGENT, value=BASELINE_AGENT),
22                    dr.experiments.Level(name=PLANNER_AGENT, value=PLANNER_AGENT),
23                ),
24            ),
25        ),
26        problem_ids=(PROBLEM_ID,),
27        run_budget=dr.experiments.RunBudget(replicates=10, parallelism=1, max_runs=20),
28        output_dir=OUTPUT_DIR,
29    )
30
31    # Build the condition table and bind each agent id to the same callable. The
32    # callable reads the condition to choose the scripted transition policy.
33    conditions = dr.experiments.build_design(study)
34    results = dr.experiments.run_study(
35        study,
36        conditions=conditions,
37        agent_bindings={BASELINE_AGENT: _agent_run, PLANNER_AGENT: _agent_run},
38        checkpoint=False,
39        show_progress=False,
40    )
41
42    # Export the run history as canonical artifacts. From this point on, the
43    # analysis code works from files rather than the in-memory run objects.
44    artifacts = dr.experiments.export_analysis_tables(
45        study,
46        conditions=conditions,
47        run_results=results,
48        output_dir=study.output_dir / "analysis",
49        validate_with_analysis_package=True,
50    )
51
52    # Fit one Markov chain per treatment from event sequences, then compare the
53    # transition matrices directly from the same event artifact.
54    chains = dr.analysis.fit_markov_chains_from_artifacts(
55        artifacts["events.csv"],
56        condition_column="agent_id",
57        session_column="run_id",
58    )
59    comparison = dr.analysis.compare_markov_chains_from_artifacts(
60        artifacts["events.csv"],
61        condition_column="agent_id",
62        left_condition=PLANNER_AGENT,
63        right_condition=BASELINE_AGENT,
64        session_column="run_id",
65    )
66
67    # Outcome metrics use the same artifact-first path, so process analysis and
68    # score summaries come from a single exported contract.
69    metric_rows = dr.analysis.build_condition_metric_table_from_artifacts(
70        artifacts["events.csv"],
71        metric=PRIMARY_METRIC,
72        condition_column="agent_id",
73    )
74    validation = dr.analysis.validate_experiment_events(artifacts["events.csv"])
75
76    means = _means_by_condition(metric_rows)
77
78    # The printout is intentionally brief: headline process comparison, outcome
79    # means, and the artifact directory for deeper inspection.
80    print("Long agent Markov comparison:", study.study_id)
81    print("Problem:", problem.metadata.title)
82    print("Actions per run:", ACTION_COUNT)
83    print("Runs:", len(results))
84    print("Event rows valid:", validation.is_valid, f"(rows={validation.n_rows})")
85    print("States:", len(chains[PLANNER_AGENT].states))
86    print("Mean primary_outcome:")
87    for agent_id in (BASELINE_AGENT, PLANNER_AGENT):
88        print(f"- {agent_id}: {means[agent_id]:.3f}")
89    print("Transition matrix delta:", f"{comparison.estimate:.4f}")
90    if comparison.p_value is not None:
91        print("Transition matrix p-value:", f"{comparison.p_value:.4f}")
92    print("Artifacts directory:", artifacts["events.csv"].parent)
93
94

Selected Output#

The complete output also names the problem, reports each condition’s mean outcome and transition-matrix p-value, and gives the artifact directory. These lines capture the main process-comparison checks:

Long agent Markov comparison: long_agent_markov_comparison
Actions per run: 30
Runs: 20
Event rows valid: True (rows=600)
States: 6
Transition matrix delta: 0.8627

The exact p-value is deterministic for this seeded example. In a real study, define the action vocabulary and coding reliability before interpreting process differences.

Next, use Compose The Libraries: Analyze A Partial Factorial Study when predictors and interactions are more important than transition structure.