API#

The top-level package exports a curated public API suitable for notebook/script use:

Study
Factor
FactorKind
Level
Constraint
Condition
Block
RunBudget
SeedPolicy
ProblemPacket
RecipeStudyConfig
ComparisonStudyConfig
UnivariateComparisonConfig
BivariateComparisonConfig
StrategyComparisonConfig
PromptFramingConfig
OptimizationBenchmarkConfig
AgentArchitectureComparisonConfig
GrammarScaffoldConfig
HumanVsAgentProcessConfig
DiversityAndExplorationConfig
Hypothesis
OutcomeSpec
AnalysisPlan
RunSpec
RunResult
BenchmarkBundle
ideation_bundle
optimization_bundle
grammar_problem_bundle
human_vs_agent_bundle
build_univariate_comparison_study
build_bivariate_comparison_study
build_strategy_comparison_study
build_prompt_framing_study
build_optimization_benchmark_study
build_agent_architecture_comparison_study
build_grammar_scaffold_study
build_human_vs_agent_process_study
build_diversity_and_exploration_study
build_design
generate_doe
materialize_conditions
run_study
resume_study
export_analysis_tables
render_markdown_summary
render_methods_scaffold
render_codebook
render_significance_brief
write_markdown_report
validate_study

Public API exports for design-research-experiments.

class design_research_experiments.AgentArchitectureComparisonConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None)[source]#

Bases: RecipeStudyConfig

Overrides for the agent architecture comparison recipe.

class design_research_experiments.AnalysisPlan(analysis_plan_id, hypothesis_ids, tests, outcomes=(), covariates=(), random_effects=(), filters=<factory>, multiple_comparison_policy='none', plots=(), export_tables=(), notes='')[source]#

Bases: object

Mapping from hypotheses to downstream analyses.

analysis_plan_id#

covariates#

export_tables#

filters#

hypothesis_ids#

multiple_comparison_policy#

notes#

outcomes#

plots#

random_effects#

tests#

class design_research_experiments.BenchmarkBundle(bundle_id, name, description, problem_ids, agent_specs, metadata=<factory>)[source]#

Bases: object

Named bundle of agent/problem defaults for a study recipe.

agent_specs#

bundle_id#

description#

metadata#

name#

problem_ids#

class design_research_experiments.BivariateComparisonConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None, comparison_factor=None, secondary_factor=None)[source]#

Bases: ComparisonStudyConfig

Overrides for the bivariate comparison recipe.

class design_research_experiments.Block(name, levels, metadata=<factory>)[source]#

Bases: object

Blocking structure for design-of-experiments materialization.

levels#

metadata#

name#

class design_research_experiments.ComparisonStudyConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None, comparison_factor=None, secondary_factor=None)[source]#

Bases: RecipeStudyConfig

Overrides for comparison-study recipe scaffolds.

comparison_factor#

secondary_factor#

class design_research_experiments.Condition(condition_id, factor_assignments, block_assignments, metadata=<factory>, admissible=True, constraint_messages=<factory>)[source]#

Bases: object

One realized treatment combination.

Parameters:

condition_id – Stable condition ID.
factor_assignments – Materialized factor assignments.
block_assignments – Materialized block assignments.
metadata – Optional metadata payload.
admissible – Constraint admissibility flag.
constraint_messages – Constraint warning/error messages.

admissible#

block_assignments#

condition_id#

constraint_messages#

factor_assignments#

metadata#

class design_research_experiments.Constraint(constraint_id, description, expression=None, callable_ref=None, severity=ConstraintSeverity.ERROR)[source]#

Bases: object

Admissibility rule over factor and block assignments.

Parameters:

constraint_id – Stable identifier for this constraint.
description – Human-readable description.
expression – Optional safe expression string.
callable_ref – Optional module:callable reference.
severity – Whether violation should fail or warn.

callable_ref#

constraint_id#

description#

evaluate(factors, blocks)[source]#: Return whether this constraint passes for one assignment.

expression#

severity#

class design_research_experiments.DiversityAndExplorationConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None)[source]#

Bases: RecipeStudyConfig

Overrides for the diversity and exploration recipe.

class design_research_experiments.Factor(name, description, kind=FactorKind.MANIPULATED, levels=(), dtype=None, default=None, metadata=<factory>)[source]#

Bases: object

Definition of one experimental factor.

Parameters:

name – Stable factor identifier.
description – Human-readable description.
kind – Factor type.
levels – Allowed level set.
dtype – Optional value type hint.
default – Optional default level value.
metadata – Optional metadata payload.

default#

description#

dtype#

iter_values()[source]#: Return all admissible values for cartesian materialization.

kind#

levels#

metadata#

name#

class design_research_experiments.FactorKind(*values)[source]#

Bases: StrEnum

Classification for an experimental factor.

BLOCKED = 'blocked'#

MANIPULATED = 'manipulated'#

MEASURED = 'measured'#

NUISANCE = 'nuisance'#

class design_research_experiments.GrammarScaffoldConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None)[source]#

Bases: RecipeStudyConfig

Overrides for the grammar scaffold recipe.

class design_research_experiments.HumanVsAgentProcessConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None)[source]#

Bases: RecipeStudyConfig

Overrides for the human-vs-agent process recipe.

class design_research_experiments.Hypothesis(hypothesis_id, label, statement, kind=HypothesisKind.EFFECT, independent_vars=(), dependent_vars=(), moderators=(), mediators=(), contrast=None, direction=HypothesisDirection.DIFFERENT, minimum_effect_of_interest=None, linked_analysis_plan_id=None, notes='')[source]#

Bases: object

Machine-readable research claim.

contrast#

dependent_vars#

direction#

hypothesis_id#

independent_vars#

kind#

label#

linked_analysis_plan_id#

mediators#

minimum_effect_of_interest#

moderators#

notes#

statement#

class design_research_experiments.Level(name, value, label=None, metadata=<factory>)[source]#

Bases: object

One admissible level/value for a factor.

Parameters:

name – Stable level identifier.
value – Encoded level value.
label – Optional display label.
metadata – Optional metadata payload.

label#

metadata#

name#

value#

class design_research_experiments.OptimizationBenchmarkConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None)[source]#

Bases: RecipeStudyConfig

Overrides for the optimization benchmark recipe.

class design_research_experiments.OutcomeSpec(name, source_table, column, aggregation, derivation=None, primary=False, expected_type='float', missing_data_policy='drop', description='')[source]#

Bases: object

Outcome definition mapped to source tables and derivations.

aggregation#

column#

derivation#

description#

expected_type#

missing_data_policy#

name#

primary#

source_table#

class design_research_experiments.ProblemPacket(problem_id, family, brief, payload=<factory>, metadata=<factory>, evaluator=None)[source]#

Bases: object

Normalized executable problem payload.

brief#

evaluator#

family#

metadata#

payload#

problem_id#

class design_research_experiments.PromptFramingConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None)[source]#

Bases: RecipeStudyConfig

Overrides for the prompt framing recipe.

class design_research_experiments.RecipeStudyConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None)[source]#

Bases: object

Shared typed overrides for recipe study construction.

Any field set to None keeps the recipe default. Any field set to a non-None value replaces that section of the study definition wholesale.

agent_specs#

analysis_plans#

authors#

blocks#

bundle#

constraints#

description#

design_spec#

factors#

hypotheses#

notes#

outcomes#

output_dir#

primary_outcomes#

problem_ids#

provenance_metadata#

rationale#

run_budget#

secondary_outcomes#

seed_policy#

study_id#

tags#

title#

class design_research_experiments.RunBudget(replicates=1, max_runs=None, parallelism=1, fail_fast=False)[source]#

Bases: object

Execution budget controls for one study.

Parameters:

replicates – Number of run replicates for each unit.
max_runs – Optional upper bound on total runs.
parallelism – Local worker count.
fail_fast – Stop after first failure when True.

fail_fast#

max_runs#

parallelism#

replicates#

class design_research_experiments.RunResult(run_id, status, outputs=<factory>, metrics=<factory>, evaluator_outputs=<factory>, cost=0.0, latency=0.0, trace_refs=<factory>, artifact_refs=<factory>, error_info=None, provenance_info=<factory>, observations=<factory>, run_spec=None, started_at=None, ended_at=None)[source]#

Bases: object

Normalized result bundle for one run.

artifact_refs#

cost#

ended_at#

error_info#

evaluator_outputs#

latency#

metrics#

observations#

outputs#

provenance_info#

run_id#

run_spec#

started_at#

status#

trace_refs#

class design_research_experiments.RunSpec(run_id, study_id, condition_id, problem_id, replicate, seed, agent_spec_ref, problem_spec_ref, execution_metadata=<factory>)[source]#

Bases: object

One executable run specification.

agent_spec_ref#

condition_id#

execution_metadata#

problem_id#

problem_spec_ref#

replicate#

run_id#

seed#

study_id#

class design_research_experiments.SeedPolicy(base_seed=0, strategy='condition_replicate', per_run_offset=9973)[source]#

Bases: object

Deterministic seed policy for run generation.

Parameters:

base_seed – Study-wide base seed.
strategy – Seed derivation strategy name.
per_run_offset – Numeric offset mixed into each run seed.

base_seed#

derive_seed(study_id, condition_id, replicate, salt='')[source]#: Derive one deterministic per-run seed.

per_run_offset#

strategy#

class design_research_experiments.StrategyComparisonConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None, comparison_factor=None, secondary_factor=None)[source]#

Bases: ComparisonStudyConfig

Overrides for the packaged-problem strategy comparison recipe.

class design_research_experiments.Study(study_id, title, description, authors=(), rationale='', tags=(), hypotheses=(), factors=(), blocks=(), constraints=(), design_spec=<factory>, outcomes=(), analysis_plans=(), run_budget=<factory>, seed_policy=<factory>, output_dir=None, provenance_metadata=<factory>, notes='', problem_ids=(), agent_specs=(), primary_outcomes=(), secondary_outcomes=())[source]#

Bases: object

Top-level experiment definition.

agent_specs#

analysis_plans#

authors#

blocks#

constraints#

description#

design_spec#

factors#

classmethod from_dict(payload)[source]#: Construct a study from a loose mapping payload.

classmethod from_json(path)[source]#: Load a study from JSON.

classmethod from_yaml(path)[source]#: Load a study from YAML.

hypotheses#

notes#

outcomes#

output_dir#

primary_outcomes#

problem_ids#

provenance_metadata#

rationale#

run_budget#

secondary_outcomes#

seed_policy#

study_id#

tags#

title#

to_dict()[source]#: Serialize the study to a stable JSON/YAML-friendly mapping.

to_json(path)[source]#: Write the study definition to JSON.

to_yaml(path)[source]#: Write the study definition to YAML.

class design_research_experiments.UnivariateComparisonConfig(study_id=None, title=None, description=None, authors=None, rationale=None, tags=None, hypotheses=None, factors=None, blocks=None, constraints=None, design_spec=None, outcomes=None, analysis_plans=None, run_budget=None, seed_policy=None, output_dir=None, provenance_metadata=None, notes=None, problem_ids=None, agent_specs=None, primary_outcomes=None, secondary_outcomes=None, bundle=None, comparison_factor=None, secondary_factor=None)[source]#

Bases: ComparisonStudyConfig

Overrides for the univariate comparison recipe.

design_research_experiments.build_agent_architecture_comparison_study(config=None)[source]#: Build a study comparing agent architecture choices across prompt difficulty.

design_research_experiments.build_bivariate_comparison_study(config=None)[source]#: Build a two-factor comparison study scaffold over packaged problems.

design_research_experiments.build_design(study, design_spec=None)[source]#: Build study conditions for the configured design.

design_research_experiments.build_diversity_and_exploration_study(config=None)[source]#: Build a study evaluating diversity and exploration outcomes.

design_research_experiments.build_grammar_scaffold_study(config=None)[source]#: Build a study comparing unconstrained and grammar-guided generation.

design_research_experiments.build_human_vs_agent_process_study(config=None)[source]#: Build a study comparing human-only, AI-assisted, and hybrid teams.

design_research_experiments.build_optimization_benchmark_study(config=None)[source]#: Build a benchmark study for optimization generalization and learning effects.

design_research_experiments.build_prompt_framing_study(config=None)[source]#: Build an ideation study with framing and prompt manipulation.

design_research_experiments.build_strategy_comparison_study(config=None)[source]#: Build a packaged-problem strategy comparison study scaffold.

design_research_experiments.build_univariate_comparison_study(config=None)[source]#: Build a one-factor comparison study scaffold over packaged problems.

design_research_experiments.export_analysis_tables(study, *, conditions, run_results, output_dir=None, include_sqlite=False, validate_with_analysis_package=False)[source]#: Export canonical analysis tables aligned with downstream workflows.

design_research_experiments.generate_doe(*, kind, factors, n_samples=None, seed=0, center_points=0, replicates=1, randomize=True, block_randomization_key=None)[source]#: Generate and summarize a DOE table for migration from drcutils-style workflows.

design_research_experiments.grammar_problem_bundle()[source]#: Return a default grammar-scaffold benchmark bundle.

design_research_experiments.human_vs_agent_bundle()[source]#: Return a default human-vs-agent teaming benchmark bundle.

design_research_experiments.ideation_bundle()[source]#: Return a default ideation-focused benchmark bundle.

design_research_experiments.materialize_conditions(factors, blocks=None, constraints=None, *, seed=None, randomize=False, counterbalance=False)[source]#: Materialize admissible conditions from factors, blocks, and constraints.

design_research_experiments.optimization_bundle()[source]#: Return a default optimization-focused benchmark bundle.

design_research_experiments.render_codebook(study, conditions)[source]#: Render a simple codebook of factors, blocks, and condition IDs.

design_research_experiments.render_markdown_summary(study, run_results)[source]#: Render a compact markdown summary for one completed study.

design_research_experiments.render_methods_scaffold(study)[source]#: Render a methods-section scaffold for manuscript drafting.

design_research_experiments.render_significance_brief(analysis_rows)[source]#: Render a short significance/effect-size brief from analysis outputs.

design_research_experiments.resolve_problem(problem_spec_ref, *, registry=None)[source]#: Resolve a problem reference into a normalized packet.

design_research_experiments.resume_study(study, *, conditions=None, agent_bindings=None, problem_registry=None, parallelism=None, checkpoint=True, include_sqlite=False, show_progress=None)[source]#: Resume a study from checkpointed run results.

design_research_experiments.run_study(study, *, conditions=None, agent_bindings=None, problem_registry=None, parallelism=None, dry_run=False, resume=False, checkpoint=True, include_sqlite=False, show_progress=None)[source]#: Run a study end-to-end and export canonical artifacts.

design_research_experiments.validate_study(study)[source]#: Validate cross-object references and study consistency.

design_research_experiments.write_markdown_report(output_dir, filename, content)[source]#: Write markdown report content into the study artifacts directory.