Analyst Toolkit

Run each module interactively inside a Jupyter notebook.

Example

from analyst_toolkit.m02_validation.run_validation_pipeline import run_validation_pipeline
from analyst_toolkit.m00_utils.config_loader import load_config
from analyst_toolkit.m00_utils.load_data import load_csv

# --- Load config and data ---
config = load_config("config/validation_config_template.yaml")
df = load_csv("path/to/your/data.csv")

# --- Extract global settings ---
notebook_mode = config.get("notebook", True)
run_id = config.get("run_id", "demo_run")

# --- Run Validation Module ---
df_validated = run_validation_pipeline(
    config=config,
    df=df,
    notebook=notebook_mode,
    run_id=run_id
)

Modules render dashboards inline if notebook: true is set in the YAML config.

See 📗 Notebook Usage Guide for a full breakdown

Run the full pipeline interactively inside a Jupyter notebook.

Example

from analyst_toolkit.run_toolkit_pipeline import run_full_pipeline

final_df = run_full_pipeline(config_path="config/run_toolkit_config.yaml")

Each module reads its own YAML config file, with optional global overrides in config/run_toolkit_config.yaml. Example:

# --- Global Run Settings ---
run_id: "CLI_2_QA"
notebook: false

# --- Pipeline Entry Point ---
pipeline_entry_path: "data/raw/synthetic_penguins_v3.5.csv"

modules:
  diagnostics:
    run: true
    config_path: "config/diag_config_template.yaml"

  validation:
    run: true
    config_path: "config/validation_config_template.yaml"

See 📗 Notebook Usage Guide for a full breakdown

make pipeline                              # uses config/run_toolkit_config.yaml
make pipeline CONFIG=config/my_config.yaml # custom config
# or directly:
python -m analyst_toolkit.run_toolkit_pipeline --config config/run_toolkit_config.yaml

For full structure and explanation, 📘 Read the Full Usage Guide

Why build a toolkit for analysts?

I built the Analyst Toolkit to eliminate the most frustrating part of the analytics workflow — wasting hours on boilerplate cleaning when we should be exploring, validating, and learning. This system gives you:

• A one-stop first-pass QA and cleaning run, fully executable in a single notebook
• Total modularity — run stage by stage or all at once
• YAML-driven control over everything from null handling to audit thresholds

Every step leaves behind artifacts: dashboards, exports, warnings, checkpoints. You don't just run the pipeline — you see it working. You know what changed, where it changed, and what the implications are downstream. This is auditable automation — and the current dashboard set is meant to be reviewed, not just generated.

It is overbuilt in the ways that matter: transparency, reproducibility, trust. It's designed for team collaboration, for portfolio projects, for production QA. It's for your current self — and your future self — when you need to revisit a workflow six months from now.

The system is human readable and YAML-driven — for your team, your stakeholders, and yourself.

This toolkit is developed and tested using the Dirty Birds v3.5 dataset — a fully synthetic recreation of the Palmer Penguins dataset, purposefully enriched with ambiguity, anomalies, and missing data. The dataset is generated using penguin_synthetic_data_generator.py, a synthetic data generator that simulates viable research data and injects realistic biological variance and field collection noise for robust QA testing.

🐧 Features include:

• Categorical anomalies (typos, whitespace, & swaps)
• Numeric outliers and skew (both in error and in biological boundaries)
• Nullable fields in both wide and narrow formats
• Simulated noise to match real-world field data collection

v0.5.0 — MCP Platform Upgrade

• Input Ingest + Retry Semantics: Added register_input and upload_input as first-class MCP flows with canonical input_ids, stable idempotency keys, and conflict-safe retry handling.
• Session Lifecycle Management: Added manage_session with retention policy visibility, on-demand config inspection, fork/rebind support, and an optional SQLite-backed durable session store.
• Artifact Access + Delivery: Added read_artifact, hardened local artifact server routing, and stabilized same-run remote artifact identities on retry.
• Certification + Contract Alignment: infer_configs, validation, and final_audit now align inferred rules to transformed session state instead of failing on obvious runtime drift.
• Operator Hardening: Improved trust boundaries around artifact serving, SQLite state paths, input registry/session behavior, and stdio troubleshooting guidance.

v0.4.4 — Full Dashboard Rollout

• Complete Dashboard Surface: All ten pipeline modules now produce standalone exportable HTML dashboards — cockpit, diagnostics, validation, normalization, duplicates, outlier detection, outlier handling, imputation, auto-heal, and data dictionary.
• Cockpit Hub: A unified landing page links every module dashboard for a run into a single navigable session view with run metadata, artifact links, and health score.
• Dashboard Renderer Modularization: dashboard_html.py refactored from a monolith into a composable renderer stack (dashboard_core, dashboard_shared, per-module renderers) — easier to review, extend, and test.
• Local Artifact Server: Optional server turns cockpit and module artifact file references into browser-openable links for local development and review.
• Data Dictionary Artifacts: Standalone data dictionary generation, surfaced as an MCP tool and as an HTML artifact with column-level schema, type, and sample coverage.
• MCP Template + Resource Inventory: resources/list and resources/read now expose the full template and resource inventory — quickstart, playbook, capability catalog, module YAML templates — directly to MCP clients and agents.
• Export Destination Routing: Configurable artifact routing with explicit GCS, local, and (stubbed) Google Drive destinations.
• Observability + Auth Hardening: Structured request lifecycle logging (ANALYST_MCP_STRUCTURED_LOGS), /metrics and /ready endpoints, and optional bearer token auth (ANALYST_MCP_AUTH_TOKEN).
• CI Hardening: Tightened lint, mypy, and test gates; CodeRabbit review workflow added to PR process.

v0.4.0 — The Cockpit Upgrade

• State Management: Introduced StateStore for in-memory DataFrame persistence between tool calls via session_id.
• Data Health Score: Every run now generates a weighted 0-100 score (Completeness, Validity, Uniqueness, Consistency).
• Healing Ledger: Persistent JSON/GCS history tracking every transformation made during a run.
• Golden Templates: Example templates tuned for typical fraud/migration/compliance patterns (bundled in the image under config/golden_templates/).
• Autonomous Tools: Added auto_heal (one-click cleaning) and drift_detection (schema/statistical comparison).
• Configuration Intelligence: Added get_config_schema to return JSON Schemas for every module.

v0.3.0

• MCP Server: New analyst_toolkit/mcp_server/ package exposes all toolkit modules as MCP tools over JSON-RPC 2.0 (HTTP /rpc) and stdio transport.
• HTML Reports: All modules can emit self-contained single-page HTML reports.
• Docker / GHCR: Image published to ghcr.io/g-schumacher44/analyst-toolkit-mcp on every push to main.
• CI + Quality: GitHub Actions: ruff lint, mypy, pytest, Docker build + GHCR push on main.

v0.2.1

• Normalization · Datetime parsing: Multi-format support, strict mode, dayfirst/yearfirst/utc options.
• Exports · Excel date stability: Explicit date formats for cross-platform rendering.
• Duplicates · Subset-focused clusters: Dashboard now focuses on chosen subset_columns for clarity.

v0.2.0

• Standardized Configuration Handling: All modules now intelligently parse their own configuration blocks.
• Simplified Module API: Runners accept the full config object — no manual unpacking needed.

v0.1.3

• Refactored Duplicates Module (M04) with correct flag/remove modes and decoupled detection logic.

v0.1.2

• Core module scaffolding complete (M01–M10), full pipeline execution, inline dashboards, joblib checkpointing.

📦 src/                                    # Source root
│
├── analyst_toolkit/                       # 🔧 Main toolkit package
│   ├── run_toolkit_pipeline.py            # CLI + notebook entrypoint
│   │
│   ├── m00_utils/                         # Shared utilities
│   │   ├── config_loader.py               # YAML config loading and merging
│   │   ├── load_data.py                   # CSV/parquet ingestion
│   │   ├── export_utils.py                # Excel + HTML export helpers
│   │   ├── report_generator.py            # Self-contained HTML report builder
│   │   ├── scoring.py                     # Data health scoring (0-100)
│   │   ├── rendering_utils.py             # Shared display/rendering helpers
│   │   ├── data_viewer.py                 # DataFrame preview utilities
│   │   ├── plot_viewer.py                 # Inline plot display
│   │   └── plot_viewer_comparison.py      # Before/after comparison plots
│   │
│   ├── m01_diagnostics/                   # Data profiling and structural diagnostics
│   ├── m02_validation/                    # Schema validation and certification gate
│   ├── m03_normalization/                 # Data cleaning and standardization
│   ├── m04_duplicates/                    # Duplicate detection and removal
│   ├── m05_detect_outliers/               # Outlier detection (IQR, z-score)
│   ├── m06_outlier_handling/              # Outlier imputation or transformation
│   ├── m07_imputation/                    # Missing data imputation
│   ├── m08_visuals/                       # Plotting utilities and dashboard rendering
│   │   ├── comparison_plots.py            # Before/after visual comparisons
│   │   ├── distributions.py               # Distribution and histogram plots
│   │   └── summary_plots.py               # Summary/overview charts
│   │
│   ├── m10_final_audit/                   # Final audit, edits, and pipeline certification
│   │
│   └── mcp_server/                        # MCP server — exposes toolkit as tools over JSON-RPC/stdio
│       ├── server.py                      # FastAPI /rpc dispatcher + stdio transport
│       ├── io.py                          # GCS/parquet/CSV data loading + report upload
│       ├── config_models.py               # Pydantic models for typed config validation
│       ├── schemas.py                     # TypedDicts and JSON Schema for tool I/O
│       ├── registry.py                    # Tool self-registration and dispatch
│       ├── state.py                       # StateStore — in-memory session management
│       ├── templates.py                   # Golden template loader and resolver
│       └── tools/                         # Self-registering tool modules (one per toolkit module)
│
├── 🧪 notebooks/                          # Interactive tutorial notebooks (modular & full run)
│
├── ⚙️ config/                             # YAML configuration files (one per module + full run)
│   └── golden_templates/                  # Best-practice configs for Fraud, Migration, Compliance
│
├── 📂 data/
│   ├── raw/                               # Original input datasets
│   ├── processed/                         # Final certified outputs (.csv)
│   └── features/                          # Optional engineered features
│
├── 📤 exports/
│   └── sample/                            # Sample HTML dashboards, reports, plots, and cleaned output
│
├── tests/                                 # Pytest test suite (MCP smoke, unit tests)
├── resource_hub/                          # Reference, guidebooks, documentation
├── Makefile                               # Common dev and ops commands
├── pyproject.toml                         # Build config and optional extras
├── environment.yaml                       # Conda environment definition
├── requirements-mcp.txt                   # MCP server pip requirements
├── Dockerfile.mcp                         # MCP server container
└── docker-compose.mcp.yml                 # Docker Compose for local MCP server