feat: add execution graph builder plan with reference implementation

[johnnygreco]

Experimenting with this plan-first approach!

Main goal is to build out a graph of all the cells that need to be generated along with their dependencies. The idea would be to have an execution engine fetch work from this graph when it is ready.

• plans/async-engine/execution_graph_builder.plan.md
• plans/async-engine/explore_execution_graph.ipynb - Interactive notebook exploring the implementation

Part of #260
Closes #267

NOTE: The reference implementation is just meant to help build some intuition for what a concrete implementation might look like.

📋 Summary

This PR introduces the Execution Graph Builder plan document that designs a memory-efficient graph-based execution framework for async dataset generation. A reference implementation is included to build intuition and validate the design decisions—the plan document should be the primary focus for review.

🎯 Focus: The Plan Document

The core of this PR is plans/async-engine/execution_graph_builder.plan.md which details:

• Hybrid graph representation: O(C) memory for millions of records (cell nodes computed on-demand)
• Generator execution traits: START, CELL_BY_CELL, ROW_STREAMABLE, BARRIER inferred from generator properties
• Cell-level dependency resolution: Enables fine-grained async execution
• Thread-safe completion tracking: Separate trackers for asyncio vs thread pool execution

🔄 Changes

✨ Added

Plan & Exploration:

• plans/async-engine/execution_graph_builder.plan.md - Detailed design document for the execution graph framework
• plans/async-engine/explore_execution_graph.ipynb - Interactive notebook exploring the implementation

Reference Implementation (in packages/data-designer-engine/src/data_designer/engine/execution_graph/):

• node_id.py - CellNodeId, BarrierNodeId, and NodeId type alias
• traits.py - ExecutionTraits Flag enum
• column_descriptor.py - Column metadata with traits and dependencies
• graph.py - ExecutionGraph with hybrid representation
• builder.py - GraphBuilder factory with trait inference
• completion.py - CompletionTracker and ThreadSafeCompletionTracker

Generator Properties:

• Added is_row_streamable property to ColumnGenerator base class and overrides in subclasses

Tests (in packages/data-designer-engine/tests/engine/execution_graph/):

• Comprehensive test suite covering graph building, node iteration, dependency resolution, and completion tracking

🔧 Changed

• Minor updates to ModelConfig and schema transform processor

🔍 Attention Areas

⚠️ Reviewers: Please pay special attention to the following:

• execution_graph_builder.plan.md - Primary review target: Does the design adequately address async execution requirements?
• graph.py - Core hybrid representation logic
• builder.py#L175-L210 - Trait inference logic (plugin-compatible approach)

---

🤖 Generated with AI