feat: Add NpyCodec for lazy-loading numpy arrays

[dimitri-yatsenko]

Summary

This PR introduces the <npy@> codec for schema-addressed numpy array storage with lazy loading capabilities within the Object-Augmented Schema (OAS).

Key Features

• Lazy loading: Inspect array shape and dtype without downloading data
• Memory mapping: Random access to large arrays via mmap_mode parameter
• NumPy integration: Transparent array operations via __array__ protocol
• Safe bulk fetch: Returns NpyRef objects instead of downloading all arrays
• Portable format: Standard .npy files readable by NumPy, MATLAB, etc.
• Schema-addressed: Paths mirror database structure ({schema}/{table}/{pk}/{attr}.npy)

OAS Codec Strategy

The Object-Augmented Schema integrates relational tables with object storage as a single system. Codecs define how Python objects are stored.

Built-in codecs (no extra dependencies):

• <blob> / <blob@> - DataJoint legacy serialization (hash-addressed)
• <npy@> - portable numpy arrays with lazy loading (schema-addressed) ← this PR
• <object@> - files/folders (schema-addressed)
• <hash@> - raw bytes (hash-addressed)

Separate packages (optional install, additional dependencies):

• datajoint-zarr → <zarr@> (requires zarr)
• datajoint-parquet → <parquet@> (requires pyarrow)
• datajoint-tiff → <tiff@> (requires tifffile)

This keeps datajoint-python lean while allowing users to install codec packages as needed. The SchemaCodec base class introduced here enables community-contributed codecs.

OAS Addressing Schemes

Scheme	Path Derived From	Deduplication	Codecs
Hash-addressed	Content hash (MD5)	Yes	<hash@>, <blob@>, <attach@>
Schema-addressed	Schema structure	No	<object@>, <npy@>

Changes

New classes:

• SchemaCodec - Abstract base class for schema-addressed codecs
• NpyRef - Lazy reference with metadata access (shape, dtype, is_loaded)
• NpyCodec - Codec implementation using .npy format

Refactoring:

• ObjectCodec now inherits from SchemaCodec (DRY)
• Renamed is_external → is_store throughout codebase for clarity

API:

• Export SchemaCodec and NpyRef from datajoint public API

Usage Example

@schema
class Neuron(dj.Imported):
    definition = """
    -> Session
    neuron_id : int16
    ---
    activity : <npy@mystore>    # Lazy-loading array
    """

# Fetch returns NpyRef, not the array
ref = (Neuron & key).fetch1('activity')
print(ref.shape)      # (1000,) - no download!
print(ref.dtype)      # float64 - no download!

# Load when ready
array = ref.load()

# Memory-mapped for large arrays (random access without full load)
mmap = ref.load(mmap_mode='r')
slice = mmap[1000:2000]  # Only reads needed portion

# Or use directly with NumPy (auto-loads)
mean = np.mean(ref)

Memory Mapping

The mmap_mode parameter enables efficient random access to large arrays:

# Local filesystem stores: mmap directly (no copy)
arr = ref.load(mmap_mode='r')

# Remote stores (S3): downloads to cache first, then mmaps
arr = ref.load(mmap_mode='r')

Modes: 'r' (read-only), 'r+' (read-write), 'c' (copy-on-write)

Comparison: <npy@> vs <blob@>

Feature	<npy@>	<blob@>
Lazy loading	✓ shape/dtype without I/O	✗ always downloads
Memory mapping	✓ via mmap_mode	✗ not supported
OAS addressing	Schema-addressed	Hash-addressed
Format	Portable .npy	DataJoint serialization
Bulk fetch	Safe (returns refs)	Downloads everything

Test Plan

• Unit tests for NpyRef metadata access
• Unit tests for NpyRef mmap_mode (local and remote)
• Unit tests for NpyCodec validation
• Integration tests for roundtrip encode/decode
• Integration tests for lazy loading and caching
• Integration tests for NumPy function integration
• All 639 tests pass

Related

• Documentation: datajoint/datajoint-docs (pre/v2.0 branch)
  • how-to/use-npy-codec.md
  • reference/specs/npy-codec.md
  • tutorials/ephys-with-npy.ipynb

🤖 Generated with Claude Code