Datanika

License: AGPL-3.0 Python 3.12+ Built with Reflex dlt dbt-core

Open-source data pipeline platform — Extract, Load, Transform, and Orchestrate from a single UI.

Datanika combines dlt (extract + load) with dbt-core (transform) and adds visual pipeline management, scheduling, and monitoring — all in one Python-native platform.

Think Airbyte + dbt Cloud + Airflow — in one tool, self-hostable with Docker Compose.

---

Features

🔌 36 Connectors — PostgreSQL, MySQL, Oracle, MongoDB, BigQuery, Snowflake, Stripe, HubSpot, Salesforce, Kafka, S3, and more 🔄 dbt Transformations — SQL models, tests, snapshots, packages, and source freshness built in 📊 Visual Pipeline Builder — DAG editor with dependency management ⏰ Scheduling — Cron-based with APScheduler, persistent across restarts 📈 Monitoring — Run history, streaming logs, and dashboard stats 🔐 Enterprise Security — RBAC, SSO (SAML/OIDC), audit logging, encrypted credentials 🌍 9 Languages — English, German, French, Spanish, Russian, Greek, Chinese, Arabic, Serbian 🔌 REST API — Full CRUD with OpenAPI/Swagger docs, rate limiting, and scoped API keys 🤖 AI-Agent Ready — hosted + local MCP server (25 tools), /llms.txt, agent-guide.md, 5-tier capability API, compile+preview validation, typed error codes, ?wait=true, Idempotency-Key, run cancellation 🚀 Pipeline Templates — One-click starter templates (Stripe→Postgres, Postgres→BigQuery, CSV→DuckDB) with prefilled connection configs 🔔 Notifications — Slack, Telegram, email, and webhook alerts on run completion, plus an in-app notification center 📦 Self-Hostable — Single docker compose up — no Kubernetes required

---

Quick Start

Docker (recommended)

git clone https://github.com/datanika-io/datanika-core.git
cd datanika-core
cp .env.example .env
# Edit .env with your settings
docker compose up -d

App available at http://localhost:3000

Development

uv venv && source .venv/bin/activate
uv pip install -e ".[dev]"
docker compose up -d postgres redis  # infrastructure only
uv run reflex run                     # starts on :3000 + :8000

---

Releases & versioning

Datanika uses 0.x SemVer with tagged releases (v0.1.0, …). While pre-1.0, breaking changes bump the minor. There is no 1.0 yet — that will mean a committed, stable public API contract.

Self-hosting? Pin a release instead of tracking master. master is continuously deployed to our hosted app and moves several times a day.

git checkout v0.1.0                                    # source
docker pull ghcr.io/datanika-io/datanika-core:v0.1.0   # image (:latest follows master)

Every version's notes are on the Releases page. Security advisories cite the first patched release (e.g. Patched: v0.1.0), so a pinned tag tells you immediately whether you're affected.

The datanika-mcp sub-package is versioned and released independently (mcp-v* tags, published to PyPI).

---

Why Datanika?

	Datanika	Airbyte	Fivetran	dbt Cloud
Extract + Load	✅ 36 connectors	✅ 400+	✅ 500+	❌
Transformations	✅ dbt built-in	❌	❌ (add-on)	✅
Scheduling	✅ Cron + DAG	✅ Basic	✅ Basic	✅
Pipeline DAG	✅ Visual	❌	❌	❌
Self-host	✅ Docker	⚠️ Needs K8s	❌ SaaS only	❌ SaaS only
Open source	✅ AGPL-3.0	⚠️ ELv2	❌	❌
Notifications	✅ Slack/Telegram/Email/Webhook	✅	✅	✅
Pricing	Free forever	Free tier limited	~$250+/mo	~$100+/mo

---

Tech Stack

Component	Technology
Frontend	Reflex (Python → React)
Backend	Starlette (via Reflex)
Extract + Load	dlt
Transform	dbt-core
Database	PostgreSQL 16
Task Queue	Celery + Redis
Scheduling	APScheduler

---

Roadmap

• 36 connectors (databases, SaaS APIs, files, streaming)
• dbt transformations, tests, snapshots, packages
• REST API v1 with OpenAPI/Swagger and typed per-connector inline schemas
• AI-agent compatibility (/llms.txt, agent-guide, 5-tier API, golden-path loop, ?wait=true, Idempotency-Key, run cancel, MCP server)
• Pipeline templates (one-click setup)
• In-app notification center with Slack, Telegram, Email, Webhook channels
• SSO (SAML/OIDC) for Enterprise
• Usage-based billing (cloud plugin)
• 2,300+ tests across unit, security, and E2E (SQLite in-memory for speed)
• Kubernetes Helm chart — in-tree at deploy/helm/datanika/; installs the same image as the Compose path. Bundled Postgres/Redis are single-replica and not production-grade — point it at managed databases (see the chart README)
• Data lineage visualization

---

AI Agent Integration

Datanika speaks MCP, so AI agents (Claude Desktop, Claude Code, Cursor, …) can browse connections, preview data, compile transformations, and monitor runs natively. 25 tools — 17 read-only, 8 write. There are two ways in.

Hosted — nothing to install. Paste this wherever your client accepts a remote MCP server and authorize in the browser:

https://app.datanika.io/mcp

OAuth 2.1, no API key to handle. Read-only unless you grant write at consent — a client that asks for nothing gets read-only, and a pasted API key stays read-only here even if its own scopes allow writes.

Local — stdio. Published on PyPI as datanika-mcp and listed on the official MCP registry as io.datanika/datanika-mcp:

# read-only by default; add --allow-write to enable the 8 write tools
uvx datanika-mcp --url https://app.datanika.io --api-key YOUR_KEY

See datanika-mcp/README.md for per-client config snippets and the full tool list, or datanika.io/docs/mcp-server for the hosted walkthrough.

Additional agent resources:

• /llms.txt — discovery document
• /api/v1/openapi.json — OpenAPI spec with typed inline schemas
• /api/v1/meta/agent-tiers — 5-tier capability stack (JSON)
• docs/api_versioning.md — stability tiers and deprecation policy

---

Open-Core Strategy

Core platform is open-source (AGPL-3.0). Cloud version adds billing, quotas, and usage metering via the datanika-cloud plugin.

---

Links

• 🌐 Website: datanika.io
• 🚀 Cloud Platform: app.datanika.io
• 📖 Documentation: datanika.io/docs
• 🔌 Connectors: datanika.io/connectors
• 📡 API Reference: datanika.io/docs/api

---

Security

Found a vulnerability? See SECURITY.md for our disclosure policy, supported versions, and reporting instructions.

---

Contributing

We welcome contributors and design partners. Open an issue or contact [email protected].

---

License

AGPL-3.0