docs: Diagram guide with schema grouping, collapse, and Mermaid examples #139

dimitri-yatsenko · 2026-01-23T18:33:50Z

Summary

Documentation for new dj.Diagram features added in datajoint-python#1345, plus new master-part how-to guide.

New Content

Master-Part How-To Guide (NEW)

how-to/master-part.ipynb - Task-oriented guide covering:
- When to use part tables vs regular foreign keys
- Defining master and part tables with -> master
- Atomic insertion with transactions
- Querying and aggregating parts
- Computed tables with parts
- Delete patterns
Cross-linked from model-relationships, define-tables, insert-data, delete-data
Added tip box in master-part spec linking to how-to

Reference Specification

reference/specs/diagram.md - Complete API reference for dj.Diagram

How-To Guide Updates

how-to/read-diagrams.ipynb - Comprehensive guide including:
- Visual notation (shapes, colors, line styles)
- Schema grouping (automatic clustering by database schema)
- Collapse feature for high-level pipeline views
- "Expanded wins" rule with examples
- "Sandwich collapse" pattern (expanding top and bottom, collapsing middle)
- Layout direction configuration
- Mermaid output for web documentation

Key Examples Documented

# Schema grouping (automatic)
dj.Diagram(schema)  # Tables grouped by schema in dashed clusters

# Collapse for high-level views
dj.Diagram(acquisition) + dj.Diagram(processing).collapse()

# Expanded wins rule
dj.Diagram(schema).collapse() + dj.Diagram(ImportantTable)

# Layout direction
dj.config.display.diagram_direction = "LR"

# Mermaid output
print(dj.Diagram(schema).make_mermaid())

Files Changed

src/how-to/master-part.ipynb - New master-part how-to guide
src/how-to/read-diagrams.ipynb - Updated with new features
src/how-to/model-relationships.ipynb - Added link to master-part
src/how-to/define-tables.md - Added See Also section with master-part link
src/how-to/insert-data.md - Added See Also section with master-part link
src/how-to/delete-data.md - Added master-part link to See Also
src/how-to/index.md - Added master-part to Schema Design section
src/reference/specs/diagram.md - New specification doc
src/reference/specs/master-part.md - Added tip linking to how-to
src/reference/specs/index.md - Added diagram spec to listing
mkdocs.yaml - Navigation updates
All 22 notebooks re-executed with updated outputs

New documentation: - Add reference/specs/diagram.md with complete API documentation - Document direction parameter, Mermaid output, and schema grouping Updates: - Add Diagram to mkdocs.yaml navigation - Add Diagram to specs index - Update read-diagrams.ipynb with new features: - Layout direction examples - Mermaid output examples - Schema grouping examples Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add second schema (howto_analysis) with cross-schema references - Demonstrate group_by_schema=True for visualizing multi-schema pipelines - Update cleanup cell to drop both schemas Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Remove group_by_schema parameter from all examples - Document automatic grouping with Python module names as labels - Update Mermaid examples to show subgraphs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

All notebooks now show diagrams with automatic schema grouping, displaying Python module names as cluster labels. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add Collapsing Schemas section to how-to guide - Add collapse() to reference documentation - Update summary table with collapsed schema visual - Re-execute notebook with collapse examples Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Added demo_modules/ with three Python modules demonstrating cross-schema dependencies: - acquisition.py: Core tables (Lab, Subject, Session) - processing.py: Processing tables (references acquisition) - analysis.py: Analysis tables (references both modules) Updated read-diagrams.ipynb with: - Extended collapse examples using actual Python modules - Fixed variable naming conflict (howto_analysis vs analysis module) - Examples showing "expanded wins" behavior - High-level view with all modules collapsed Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Added links to: - Read Schema Diagrams (read-diagrams.ipynb) in Schema Design section - Use the <npy> Codec (use-npy-codec.md) in Object Storage section Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Tutorials index: - Added Electrophysiology with Object Storage (ephys-with-npy.ipynb) Explanation index: - Added Semantic Matching - Added What's New in 2.0 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Update demo_modules to use deferred schema activation pattern - Use core types (int16, int32, float32) instead of native types - Mark collapse feature as new in DataJoint 2.1 - Fix notebook cell ordering for schema activation/drop - Update code snippets to show core types Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Changed from dj.Diagram(module.schema) to simpler dj.Diagram(module) - Added example with two schemas collapsed - Added schema-level DAG showing all modules collapsed - Re-executed notebook with updated outputs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Re-executed notebook with fixed collapse logic. All examples now render correctly: - All three modules collapsed shows schema-level DAG - Mixed mode shows acquisition expanded, others collapsed Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Reduced code block font size to 0.75em - Added pre-wrap for long lines to prevent horizontal scrolling Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Fixed duplicate alias nodes (4 → 2 orange dots) - Fixed fresh diagrams incorrectly collapsing - All collapse examples now render correctly Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Replaced markdown code block with reference to the actual executable cell that demonstrates the "Expanded wins" rule. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Inserted executable cell right after the markdown explanation so the diagram is visible immediately. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Collapsed node now shows "howto_analysis (2 tables)" instead of ambiguous "__main__", making it clear which schema is collapsed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Demonstrates collapsing middle layers of a pipeline while keeping top (RawData) and bottom (FinalResult) expanded. The collapsed node sits between them maintaining DAG structure: RawData → (2 tables) → FinalResult Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Collapsed node now appears inside the cluster with RawData and FinalResult, maintaining proper visual flow: RawData → (2 tables) → FinalResult Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

All diagrams now use left-to-right layout by default, matching the natural data flow representation in pipelines. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Collapsed nodes now display "(N tables)" without redundant name since the cluster label already shows the schema/module name. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Create how-to/master-part.ipynb covering compositional data patterns - Add to mkdocs navigation and how-to index - Cross-link from model-relationships, define-tables, insert-data, delete-data - Add tip box in master-part spec linking to how-to Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Mark schema grouping, collapse, mermaid output, and layout direction as new features in DataJoint 2.1. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

The mermaid code block will render in the built docs via mkdocs-mermaid2. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Clarify that visual clustering is new in 2.1, not multi-schema support - Remove mermaid rendered example (mkdocs-jupyter doesn't process it) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

dimitri-yatsenko force-pushed the feat/diagram-improvements branch 2 times, most recently from 2178c9b to d4d4235 Compare January 23, 2026 19:21

dimitri-yatsenko force-pushed the feat/diagram-improvements branch from d4d4235 to f4d7c2e Compare January 23, 2026 19:28

dimitri-yatsenko and others added 22 commits January 23, 2026 13:34

docs: update for automatic schema grouping

6f5fa1b

- Remove group_by_schema parameter from all examples - Document automatic grouping with Python module names as labels - Update Mermaid examples to show subgraphs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: re-execute all notebooks with schema grouping

50cb675

All notebooks now show diagrams with automatic schema grouping, displaying Python module names as cluster labels. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: re-execute notebook with improved schema grouping

deb853c

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: add missing how-to guides to index

8e579a0

Added links to: - Read Schema Diagrams (read-diagrams.ipynb) in Schema Design section - Use the <npy> Codec (use-npy-codec.md) in Object Storage section Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: add missing files to section indices

a1b1e9f

Tutorials index: - Added Electrophysiology with Object Storage (ephys-with-npy.ipynb) Explanation index: - Added Semantic Matching - Added What's New in 2.0 Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: execute read-diagrams notebook with diagram outputs

ac87c1e

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

style: smaller code font and line wrapping

d04bdd7

- Reduced code block font size to 0.75em - Added pre-wrap for long lines to prevent horizontal scrolling Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: re-execute notebook with diagram bug fixes

5b0c09e

- Fixed duplicate alias nodes (4 → 2 orange dots) - Fixed fresh diagrams incorrectly collapsing - All collapse examples now render correctly Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: remove non-executable code block from markdown

4a42207

Replaced markdown code block with reference to the actual executable cell that demonstrates the "Expanded wins" rule. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: add executable cell for Expanded wins example

0fa7775

Inserted executable cell right after the markdown explanation so the diagram is visible immediately. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: update Expanded wins example with database schema name fix

3b6c485

Collapsed node now shows "howto_analysis (2 tables)" instead of ambiguous "__main__", making it clear which schema is collapsed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: update sandwich example with fixed cluster layout

3420e5a

Collapsed node now appears inside the cluster with RawData and FinalResult, maintaining proper visual flow: RawData → (2 tables) → FinalResult Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: re-execute all notebooks with LR layout default

02336f8

All diagrams now use left-to-right layout by default, matching the natural data flow representation in pipelines. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: update notebooks with collapsed node showing only table count

b64f657

Collapsed nodes now display "(N tables)" without redundant name since the cluster label already shows the schema/module name. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: re-execute notebooks with updated diagram collapse logic

cc0b00c

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

dimitri-yatsenko changed the title ~~WIP: docs: add Diagram specification and how-to updates~~ docs: Diagram guide with schema grouping, collapse, and Mermaid examples Jan 23, 2026

dimitri-yatsenko marked this pull request as ready for review January 23, 2026 23:57

dimitri-yatsenko requested review from MilagrosMarin and esutlie January 23, 2026 23:57

dimitri-yatsenko requested a review from ttngu207 January 23, 2026 23:57

dimitri-yatsenko and others added 5 commits January 23, 2026 21:30

docs: re-execute notebooks against PostgreSQL

4690ab8

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: add 'New in 2.1' markers to diagram features

31a56f3

Mark schema grouping, collapse, mermaid output, and layout direction as new features in DataJoint 2.1. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: add rendered mermaid example in diagram how-to

87ce3b9

The mermaid code block will render in the built docs via mkdocs-mermaid2. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

docs: fix diagram how-to version markers and remove broken mermaid

4f43fd0

- Clarify that visual clustering is new in 2.1, not multi-schema support - Remove mermaid rendered example (mkdocs-jupyter doesn't process it) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

MilagrosMarin approved these changes Jan 24, 2026

View reviewed changes

MilagrosMarin merged commit 200b0c4 into main Jan 24, 2026
1 check passed

dimitri-yatsenko mentioned this pull request Jan 24, 2026

fix: update datajoint-python ref to pre/v2.1 #140

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: Diagram guide with schema grouping, collapse, and Mermaid examples #139

docs: Diagram guide with schema grouping, collapse, and Mermaid examples #139

Uh oh!

dimitri-yatsenko commented Jan 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

docs: Diagram guide with schema grouping, collapse, and Mermaid examples #139

docs: Diagram guide with schema grouping, collapse, and Mermaid examples #139

Uh oh!

Conversation

dimitri-yatsenko commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New Content

Master-Part How-To Guide (NEW)

Reference Specification

How-To Guide Updates

Key Examples Documented

Files Changed

Related

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dimitri-yatsenko commented Jan 23, 2026 •

edited

Loading