Experiments: Test the Monty Hall Strategy#

Monty Hall is a useful standalone experiment because the study definition, treatment, outcome, and ground-truth expectation are all visible. We define stay and switch as two conditions, run 100 seeded games for each, and save a compact result artifact.

Setup#

python -m pip install design-research-experiments==0.3.0

Step 1: Define the study contract#

[1]:

import csv
import random
import warnings
from pathlib import Path
from typing import TypedDict

warnings.filterwarnings("ignore", message="IProgress not found.*")

import design_research_experiments as experiments  # noqa: E402

DOORS = ("A", "B", "C")
GAMES_PER_CONDITION = 100
SEED = 5

study = experiments.Study(
    study_id="monty-hall-simulation",
    title="Monty Hall Simulation",
    description="Compare stay and switch over seeded random games.",
    factors=(
        experiments.Factor(
            name="strategy",
            description="Decision after the host reveals a goat door.",
            kind=experiments.FactorKind.MANIPULATED,
            levels=(
                experiments.Level(name="stay", value="stay"),
                experiments.Level(name="switch", value="switch"),
            ),
        ),
    ),
    hypotheses=(
        experiments.Hypothesis(
            hypothesis_id="h1",
            label="Switching improves win rate",
            statement="Switching wins more often than staying.",
            independent_vars=("strategy",),
            dependent_vars=("win_rate",),
        ),
    ),
    outcomes=(
        experiments.OutcomeSpec(
            name="win_rate",
            source_table="runs",
            column="won",
            aggregation="mean",
            primary=True,
        ),
    ),
    analysis_plans=(
        experiments.AnalysisPlan(
            analysis_plan_id="ap1",
            hypothesis_ids=("h1",),
            tests=("simulation_summary",),
            outcomes=("win_rate",),
        ),
    ),
    design_spec=experiments.DesignSpec(kind=experiments.DesignKind.FULL_FACTORIAL, randomize=False),
    seed_policy=experiments.SeedPolicy(base_seed=SEED),
    problem_ids=("monty-hall-game",),
)
errors = experiments.validate_study(study)
print("Study valid:", not errors)
print("Hypothesis:", study.hypotheses[0].statement)

Study valid: True
Hypothesis: Switching wins more often than staying.

Step 2: Materialize one condition per strategy#

[2]:

conditions = experiments.build_design(study)
print("Conditions:", len(conditions))
for condition in conditions:
    print(f"- {condition.condition_id}: {condition.factor_assignments['strategy']}")

Conditions: 2
- cond-ecf5c1dda7b3: stay
- cond-a18b5b000ae6: switch

Step 3: Make the host and contestant rules explicit#

[3]:

def reveal_goat(prize: str, initial: str, rng: random.Random) -> str:
    """Return one admissible goat door for the host to reveal."""
    candidates = [door for door in DOORS if door != prize and door != initial]
    return str(rng.choice(candidates))


def final_choice(initial: str, revealed: str, strategy: str) -> str:
    """Resolve the contestant's final door from the assigned strategy."""
    if strategy == "stay":
        return initial
    return next(door for door in DOORS if door != initial and door != revealed)


demo_rng = random.Random(SEED)
demo_prize = "A"
demo_initial = "A"
demo_revealed = reveal_goat(demo_prize, demo_initial, demo_rng)
print("Prize:", demo_prize)
print("Initial choice:", demo_initial)
print("Host reveals:", demo_revealed)
print("Stay chooses:", final_choice(demo_initial, demo_revealed, "stay"))
print("Switch chooses:", final_choice(demo_initial, demo_revealed, "switch"))

Prize: A
Initial choice: A
Host reveals: C
Stay chooses: A
Switch chooses: B

Step 4: Run both seeded conditions#

[4]:

class SimulationRow(TypedDict):
    """One seeded strategy result."""

    strategy: str
    games: int
    wins: int
    win_rate: float
    seed: int


def simulate(strategy: str, *, games: int, seed: int) -> SimulationRow:
    """Simulate one seeded strategy condition."""
    rng = random.Random(seed)
    wins = 0
    for _ in range(games):
        prize = str(rng.choice(DOORS))
        initial = str(rng.choice(DOORS))
        revealed = reveal_goat(prize, initial, rng)
        choice = final_choice(initial, revealed, strategy)
        wins += int(choice == prize)
    return {
        "strategy": strategy,
        "games": games,
        "wins": wins,
        "win_rate": wins / games,
        "seed": seed,
    }


rows = [
    simulate(str(condition.factor_assignments["strategy"]), games=GAMES_PER_CONDITION, seed=SEED)
    for condition in conditions
]
for row in rows:
    print(f"{row['strategy']}: {row['wins']}/{row['games']} = {row['win_rate']:.2f}")

stay: 35/100 = 0.35
switch: 65/100 = 0.65

Step 5: Preserve the result as an artifact#

[5]:

output_path = Path("artifacts/tutorials/experiments_monty_hall/simulation_summary.csv")
output_path.parent.mkdir(parents=True, exist_ok=True)
with output_path.open("w", encoding="utf-8", newline="") as file_obj:
    writer = csv.DictWriter(file_obj, fieldnames=list(rows[0]))
    writer.writeheader()
    writer.writerows(rows)

stay, switch = rows
print("Observed lift:", f"{switch['win_rate'] - stay['win_rate']:.2f}")
print("Artifact:", output_path)
print(output_path.read_text(encoding="utf-8").strip())

Observed lift: 0.30
Artifact: artifacts/tutorials/experiments_monty_hall/simulation_summary.csv
strategy,games,wins,win_rate,seed
stay,100,35,0.35,5
switch,100,65,0.65,5

Interpretation#

The seeded run produces 35 stay wins and 65 switch wins. That finite result is close to the theoretical probabilities of one third and two thirds; changing the seed or run budget changes the observed counts but not the study contract.