LaminDB

Overview

LaminDB is an open-source, lineage-native lakehouse for biology. It makes datasets and models queryable, traceable, validated, reproducible, and FAIR (Findable, Accessible, Interoperable, Reusable) while storing data in open formats across local filesystems, S3, GCS, Hugging Face, SQLite, and Postgres.

Core Value Proposition:

• Queryability: Search and filter artifacts, records, runs, features, schemas, and collections
• Traceability: Track inputs, outputs, parameters, source code, and environments for notebooks, scripts, functions, and pipelines
• Validation: Curate DataFrame, AnnData, SpatialData, TileDB-SOMA, Parquet, Zarr, and other biological formats with schemas
• FAIR Compliance: Standardize annotations with Bionty-backed ontologies and custom registries
• Change management: Organize work with projects, branches, spaces, collections, and saved notes or plans

When to Use This Skill

Use this skill when:

• Managing biological datasets: scRNA-seq, bulk RNA-seq, spatial transcriptomics, flow cytometry, multi-modal data, EHR data
• Tracking computational workflows: Notebooks, scripts, functions, shell scripts, and pipeline execution (Nextflow, Snakemake, Redun)
• Curating and validating data: Schema validation, standardization, ontology-based annotation
• Working with biological ontologies: Genes, proteins, cell types, tissues, diseases, pathways (via Bionty)
• Building data lakehouses: Unified query interface across multiple datasets
• Ensuring reproducibility: Automatic versioning, lineage tracking, environment capture
• Integrating ML pipelines: Connecting with Weights & Biases, MLflow, Hugging Face, Lightning, scVI-tools
• Deploying data infrastructure: Setting up local or cloud-based data management systems
• Collaborating on datasets: Sharing curated, annotated data with standardized metadata

Core Capabilities

LaminDB provides six interconnected capability areas, each documented in detail in the references folder.

1. Core Concepts and Data Lineage

Core entities:

• Artifacts: Versioned datasets (DataFrame, AnnData, Parquet, Zarr, etc.)
• Records & ULabels: Experimental entities, typed records, and simple labels
• Collections: Versioned, immutable sets of artifacts
• Runs & Transforms: Computational lineage tracking (what code produced what data)
• Features: Typed metadata fields for annotation and querying
• Projects, Branches & Spaces: Project grouping, change management, and access boundaries

Key workflows:

• Create and version artifacts from files or Python objects
• Track notebook/script execution with ln.track() and ln.finish()
• Track function workflows with @ln.flow() and @ln.step()
• Annotate artifacts with records, ulabels, projects, and typed features
• Visualize data lineage graphs with artifact.view_lineage()
• Query by provenance (find all outputs from specific code/inputs)

Reference: references/core-concepts.md - Read this for detailed information on artifacts, records, runs, transforms, features, versioning, and lineage tracking.

2. Data Management and Querying

Query capabilities:

• Registry exploration and lookup with auto-complete
• Single record retrieval with get(), one(), one_or_none()
• Filtering with comparison operators (__gt, __lte, __contains, __startswith)
• Feature-based queries, including expression-style queries with Feature objects
• Cross-registry traversal with double-underscore syntax
• Full-text search across registries
• Advanced logical queries with ln.Q objects (AND, OR, NOT)
• Streaming large datasets without loading into memory

Key workflows:

• Browse artifacts with filters and ordering
• Query by features, creation date, creator, size, etc.
• Stream large files in chunks or with array slicing
• Organize data with hierarchical keys
• Group artifacts into collections

Reference: references/data-management.md - Read this for comprehensive query patterns, filtering examples, streaming strategies, and data organization best practices.

3. Annotation and Validation

Curation process:

1. Validation: Confirm datasets match desired schemas
2. Standardization: Fix typos, map synonyms to canonical terms
3. Annotation: Link datasets to metadata entities for queryability

Schema types:

• Flexible schemas: Validate only known columns, allow additional metadata
• Minimal required schemas: Specify essential columns, permit extras
• Strict schemas: Complete control over structure and values

Supported data types:

• DataFrames (Parquet, CSV)
• AnnData (single-cell genomics)
• MuData (multi-modal)
• SpatialData (spatial transcriptomics)
• TileDB-SOMA (scalable arrays)

Key workflows:

• Define features and schemas for data validation
• Use DataFrameCurator, AnnDataCurator, SpatialDataCurator, or TiledbsomaExperimentCurator for validation
• Standardize values with .cat.standardize()
• Map to ontologies with .cat.add_ontology()
• Save curated artifacts with schema linkage
• Query validated datasets by features

Reference: references/annotation-validation.md - Read this for detailed curation workflows, schema design patterns, handling validation errors, and best practices.

4. Biological Ontologies

Available ontologies (via Bionty):

• Genes (Ensembl), Proteins (UniProt)
• Cell types (CL), Cell lines (CLO)
• Tissues (Uberon), Diseases (Mondo, DOID)
• Phenotypes (HPO), Pathways (GO)
• Experimental factors (EFO), Developmental stages
• Organisms (NCBItaxon), Drugs (DrugBank)

Key workflows:

• Import public ontologies with bt.CellType.import_source()
• Search ontologies with keyword or exact matching
• Standardize terms using synonym mapping
• Explore hierarchical relationships (parents, children, ancestors)
• Validate data against ontology terms
• Annotate datasets with ontology records
• Create custom terms and hierarchies
• Handle multi-organism contexts (human, mouse, etc.)

Reference: references/ontologies.md - Read this for comprehensive ontology operations, standardization strategies, hierarchy navigation, and annotation workflows.

5. Integrations

Workflow managers:

• Nextflow: Track pipeline processes and outputs
• Snakemake: Integrate into Snakemake rules
• Redun: Combine with Redun task tracking
• Lightning: Persist checkpoints and training metadata

MLOps platforms:

• Weights & Biases: Link experiments with data artifacts
• MLflow: Track models and experiments
• Hugging Face: Track model fine-tuning
• scVI-tools: Single-cell analysis workflows

Storage systems:

• Local filesystem, AWS S3, Google Cloud Storage
• S3-compatible (MinIO, Cloudflare R2)
• HTTP/HTTPS endpoints (read-only)
• HuggingFace datasets

Array stores:

• TileDB-SOMA (with cellxgene support)
• DuckDB for SQL queries on Parquet files

Visualization:

• Vitessce for interactive spatial/single-cell visualization

Version control:

• Git integration for source code tracking

Reference: references/integrations.md - Read this for integration patterns, code examples, and troubleshooting for third-party systems.

6. Setup and Deployment

Installation:

• Current stable baseline: lamindb==2.5.1 (released 2026-06-01; Python >=3.10, <=3.14)
• Basic: uv pip install 'lamindb==2.5.1'
• With extras: uv pip install 'lamindb[gcp,zarr-v2,fcs]==2.5.1'
• Minimal namespace only: uv pip install 'lamindb-core==2.5.1'
• Bionty module: included in the LaminDB docs and available as uv pip install 'bionty==2.4.0'
• Optional modules: pin reviewed releases for wetlab or clinical schema modules rather than installing floating latest versions

Instance types:

• Local SQLite (development)
• Cloud storage + SQLite (small teams)
• Cloud storage + PostgreSQL (production)

Storage options:

• Local filesystem
• AWS S3 with configurable regions and permissions
• G