Compose The Libraries: Analyze A Partial Factorial Study

This tutorial samples model-size and design-task combinations instead of running their full cross product. It executes deterministic ideation traces, exports canonical events, and fits a regression directly from the artifact.

What You Will Learn

• Define a larger factor space while materializing a deliberate partial matrix.

• Bind problem IDs and model metadata to explicit conditions.

• Return normalized custom-agent metrics and event sequences.

• Fit numeric and categorical predictors without manually loading CSV tables.

• Keep regression inputs traceable to the exported experiment artifact.

Install And Run

Download partial_factorial_ideation_regression.py, open its containing folder in VS Code, and use the integrated terminal:

python -m pip install design-research==0.4.0
python partial_factorial_ideation_regression.py

Core Orchestration

The excerpt shows the user-facing run and analysis path. The complete source also defines the explicit condition matrix and deterministic participant.

def main() -> None:
    """Run a larger ideation DOE without touching the exported tables directly."""
    # Build the study separately from the condition matrix so the tutorial can
    # show both pieces of a custom design of experiments.
    study = _study()
    conditions = _partial_factorial_conditions()

    # The scripted agent keeps execution offline and deterministic, but it still
    # returns the same result shape expected from a live ideation agent.
    results = dr.experiments.run_study(
        study,
        conditions=conditions,
        agent_bindings={AGENT_ID: _ideation_agent},
        checkpoint=False,
        show_progress=False,
    )

    # Persist the standard artifacts before analysis. This keeps the example
    # aligned with a reproducible workflow where tables can be inspected later.
    artifacts = dr.experiments.export_analysis_tables(
        study,
        conditions=conditions,
        run_results=results,
        output_dir=study.output_dir / "analysis",
        validate_with_analysis_package=True,
    )

    # Fit a linear model from artifacts with both a numeric predictor and a
    # categorical task-family predictor. No user code touches the CSV tables.
    regression = dr.analysis.fit_regression_from_artifacts(
        artifacts["events.csv"],
        outcome=PRIMARY_METRIC,
        predictors=("model_size_b", "task_family"),
        categorical_predictors=("task_family",),
    )
    validation = dr.analysis.validate_experiment_events(artifacts["events.csv"])

    # Print the regression headline and validation status; the detailed rows stay
    # in the generated artifacts.
    print("Partial factorial ideation regression:", study.study_id)
    print("Conditions:", len(conditions))
    print("Runs:", len(results))
    print("Event rows valid:", validation.is_valid, f"(rows={validation.n_rows})")
    print("Regression samples:", regression.n_samples)
    print("Model size coefficient:", f"{regression.coefficients['model_size_b']:.4f}")
    print("Task family terms:", _task_terms(regression.coefficients))
    print("R2:", f"{regression.r2:.3f}")
    print("Artifacts directory:", artifacts["events.csv"].parent)

Selected Output

The script also lists the fitted task-family terms and artifact directory. These lines capture the main regression checks:

Partial factorial ideation regression: partial_factorial_ideation_regression
Conditions: 12
Runs: 24
Event rows valid: True (rows=120)
Regression samples: 24
Model size coefficient: 0.0098
R2: 0.992

Use an explicit partial matrix only when its estimable effects match the research question. For generated fractional-factorial or Latin-hypercube designs, use the DOE helpers owned by design-research-experiments.

Continue to Prompt Strategy Comparison Walkthrough to replace deterministic participant logic with a managed local model while preserving the same artifact boundary.