design-research-problems#

The benchmark-task layer for reproducible design research.

What This Library Does#

design-research-problems owns structured task definitions, metadata, statements, evaluators, and packaged assets across five ProblemKind values: text, decision, optimization, grammar, and MCP. The ideation catalog is a curated subset of the text family.

Stable problem metadata, packaged statements, and explicit family APIs are core features. They make benchmarks easier to compare across agents, experiments, and downstream analyses.

CI Coverage Examples Passing API in Examples Docs PyPI Version Python Versions

Quality Signals#

  • Coverage reports total line coverage for the default deterministic test suite; CI requires at least 95%.

  • Examples Passing reports checked-in example scripts that execute successfully in the examples workflow.

  • API in Examples reports curated top-level __all__ exports referenced by runnable examples. N/N means every supported top-level export appears in at least one example, and CI requires 100%.

Run make coverage, make examples-test, and make examples-coverage to reproduce these checks locally.

Highlights#

  • Packaged benchmark families for text, decision, optimization, grammar, and MCP-backed workflows

  • A linked ideation metadata catalog within the text family

  • Stable problem metadata and reusable family-specific APIs

  • A study-facing integration seam in design_research_problems.integration for orchestration layers

  • Explicit downstream metadata and evaluation contracts for experiments and analysis

  • Catalog entry points for browsing and loading packaged problems

  • Runnable examples spanning the major benchmark families

Typical Workflow#

  1. Start from a family API or a catalog entry point.

  2. Load a packaged problem and inspect its metadata, statement, and structured inputs.

  3. Let the experiments layer resolve the problem through design_research_problems.integration.resolve_problem_binding(...), invoke an agent, and preserve benchmark metadata.

  4. Capture outputs against the downstream metadata contract for comparison.

  5. Rejoin benchmark context in downstream analysis and reporting.

Note

New here? Follow Guides for the shared install → quickstart → concepts/workflow → examples → API path. Use Catalog Guide when you are ready to choose among packaged tasks.

Guides#

Learn the family model, setup flow, and benchmark-selection patterns that shape a stable problem-research pipeline.

Examples#

Browse runnable examples that show the public APIs across the major problem families.

Reference#

Look up the stable import surface, rendered catalog entry points, and optional dependency guidance for the packaged benchmark families.

Integration With The Ecosystem#

The CMU Design Research Collective design-research ecosystem is a modular set of libraries for studying human and AI design behavior.

  • design-research-agents owns executable AI participants, workflows, and tool-using reasoning patterns.

  • design-research-problems (this package) owns benchmark tasks, prompts, grammars, metadata, and evaluators.

  • design-research-experiments owns study design and coordinates artifact flows across packages.

  • design-research-analysis validates and analyzes the resulting traces, event tables, and outcomes.

Together these libraries support end-to-end design research pipelines, from study design through execution and interpretation.

The figure shows two complementary views: control responsibility and runtime artifact flow. Neither view is a package-install order. See the umbrella compatibility matrix for the component versions tested together.

Two-view diagram showing the control topology and runtime data flow across Problems, Agents, Experiments, and Analysis.

Start Here#