design-research-problems#

A library of benchmark tasks for design research.

What This Library Does#

design-research-problems provides structured design tasks spanning ideation, decision-making, optimization, grammar-based design exploration, and MCP-backed workflows. It is built for recurring research workflows where clear metadata, reusable evaluation contracts, and domain fidelity all matter.

Stable problem metadata, packaged statements, and explicit family APIs are core features. They make benchmarks easier to compare across agents, experiments, and downstream analyses.

CI Coverage Examples Passing Public API In Examples Docs PyPI Version Python Versions

Highlights#

  • Packaged benchmark families for ideation, decision, optimization, grammar, and MCP-backed workflows

  • Stable problem metadata and reusable family-specific APIs

  • A study-facing integration seam in design_research_problems.integration for orchestration layers

  • Explicit downstream metadata and evaluation contracts for experiments and analysis

  • Catalog entry points for browsing and loading packaged problems

  • Runnable examples spanning the major benchmark families

Typical Workflow#

  1. Start from a family API or a catalog entry point.

  2. Load a packaged problem and inspect its metadata, statement, and structured inputs.

  3. Hand the problem to agents or experiments through design_research_problems.integration.resolve_problem_binding(...) while preserving benchmark metadata.

  4. Capture outputs against the downstream metadata contract for comparison.

  5. Rejoin benchmark context in downstream analysis and reporting.

Note

Start with Catalog Guide if you are browsing the problem space, or Quickstart if you already know you want to load one packaged problem and enter the API quickly.

Guides#

Learn the family model, setup flow, and benchmark-selection patterns that shape a stable problem-research pipeline.

Examples#

Browse runnable examples that show the public APIs across the major problem families.

Reference#

Look up the stable import surface, rendered catalog entry points, and optional dependency guidance for the packaged benchmark families.

Integration With The Ecosystem#

The Design Research Collective maintains a modular ecosystem of libraries for studying human and AI design behavior.

  • design-research-agents implements AI participants, workflows, and tool-using reasoning patterns.

  • design-research-problems provides benchmark design tasks, prompts, grammars, and evaluators.

  • design-research-analysis analyzes the traces, event tables, and outcomes generated during studies.

  • design-research-experiments sits above the stack as the study-design and orchestration layer, defining hypotheses, factors, conditions, replications, and artifact flows across agents, problems, and analysis.

Together these libraries support end-to-end design research pipelines, from study design through execution and interpretation.

Ecosystem diagram showing experiments above agents, problems, and analysis.

Start Here#