Acknowledgements

Mcity would like to thank Amazon Web Services (AWS) for their pivotal role in providing the cloud infrastructure on which the Data Engine depends. We couldn’t have done it without their tremendous support!

Agentic Mcity Data Engine

The Agentic MCity Data Engine introduces a conversational AI layer that sits seamlessly on top of the core data engine, enabling natural language interaction with complex computer vision workflows. Built using the Model Context Protocol (MCP), the agent acts as an intelligent orchestrator that guides users through workflow configuration and execution without requiring deep technical knowledge.

On February 24, 2025, Daniel Bogdoll, a research scholar at Mcity, gave a presentation on the first release of the Mcity Data Engine in Ann Arbor, Michigan. The recording provides insight into the general architecture, its features and ecosystem integrations, and demonstrates successful data curation and model training for improved Vulnerable Road User (VRU) detection:

Key Features of the Agentic Implementation:

The Agentic Mcity Data Engine extends the Mcity Data Engine with an LLM-agnostic orchestration layer powered by the Model Context Protocol (MCP). This layer transforms the Auto Labeling workflow into structured, callable tools that can be accessed either through natural-language interaction or programmatic APIs.

Natural Language Configuration: Configure complex workflows through conversational commands instead of manually editing Python config files. The agent translates natural language requests into correct configuration settings, validates parameters, maintains context across conversation turns, and guides users through multi-step workflow setup with intelligent prompts and error prevention.

Core Components:

- User Interface : A unified entry point for interaction—users can chat via a natural-language web UI or send direct HTTP API requests from the terminal.

- Chat Server: A FastAPI service (port 8001) acting as the bridge between the user, LLM, and backend MCP services. It maintains multi-turn chat history, handles tool invocations, streams Server-Sent Event (SSE) logs, and supports both web-UI and programmatic clients.

- LLM Layer (Model-Agnostic): Connects to OpenAI, Anthropic Claude, Google Gemini, or Groq models. The LLM interprets user instructions, determines the appropriate workflow tool call, and sends structured requests back to the chat server for execution.

- MCP Server: A FastAPI-based backend (port 8000) exposing the tools that represent the Auto Labeling workflow and its shared infrastructure (dataset selection, ingestion, CVAT/Label Studio export, Voxel51 visualization).

- Data Ingestion Server: A dedicated service (port 8002) for uploading and preprocessing datasets. It supports drag-and-drop ingestion of images, videos, and annotations in COCO, YOLO, or CVAT-XML formats, automatically converting them into FiftyOne-compatible datasets. This server streams conversion logs and progress via SSE and updates datasets.yaml dynamically to register new datasets for use across workflows.

- Data Engine Core: The underlying Mcity Data Engine handling data selection, labeling, training, validation, and visualization. The agentic layer orchestrates these modules programmatically via MCP instead of relying on static configuration editing.

Online Demo: Data Selection with Embeddings

To get a first feel for the Mcity Data Engine, we provide an online demo in a Google Colab environment. We will load the Fisheye8K dataset and demonstrate the Mcity Data Engine workflow Embedding Selection. This workflow leverages a set of models to compute image embeddings which are used to determine both representative and rare samples. The dataset is then visualized in the Voxel51 UI, highlighting how often a sample was picked by the workflow.

Note that most of the Mcity Data Engine workflows require a more powerful GPU, so the possibilities within the Colab environment are limited. Other workflows may not work.

Online demo on Google Colab: Mcity Data Engine Web Demo

Local Execution

At least one GPU is required for many of the Mcity Data Engine workflows. Check the hardware setups we have tested in the Wiki. To download the repository and install the requirements run:

git clone [email protected]:mcity/mcity_data_agent.git
cd mcity_data_agent

# Use python3.12 explicitly -- plain `python3` may resolve to a newer default
# (3.13/3.14) on your system, which several pinned packages (e.g. numba==0.61.0)
# do not yet support.
python3.12 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
pip install -r agent_requirements.txt

# fiftyone requires sse-starlette<1, but mcp (via fastmcp) requires >=1.6.1 --
# no version satisfies both, so fiftyone is installed separately with --no-deps
# rather than forcing the wrong one onto the rest of the stack.
pip install "fiftyone==1.4.1" --no-deps

Login with your Weights and Biases and Hugging Face accounts:

wandb login
hf auth login

Agentic Implementation Guide:

Configuration: Create a .env file with your LLM API key.

# Choose your LLM provider (openai, claude, gemini, or groq)
LLM_PROVIDER=openai

# Add your API key
OPENAI_API_KEY=sk-...
# OR
ANTHROPIC_API_KEY=sk-ant-...
# OR
GEMINI_API_KEY=...
# OR
GROQ_API_KEY=...

Launch the Agentic Interface: Start all four components one after the other in separate terminals:

# Terminal 1: MCP Tool Server (port 8000)
python mcp_layer/mcp_server.py

# Terminal 2: Chat Server (port 8001)
uvicorn mcp_layer.chat_server:app --port 8001 --reload

# Terminal 3: Data Ingestion Server (port 8002)
uvicorn mcp_layer.ingest_server:app --host 0.0.0.0 --port 8002 --reload

# Terminal 4: Launch Web UI (port 5225)
python mcp_layer/client_chat.py --ui

The web interface will open automatically at http://localhost:5225.

Docker / AWS Deployment: agent_deployment/ does not contain a Dockerfile for running the agent itself in a local container. Instead:

• agent_deployment/Docker/ is a small Flask app (App.py + cloudFormationManager.py + redisManager.py) that provides a web form for launching a remote AWS EC2 sandbox running this repo via CloudFormation. docker-compose.yml there runs that launcher app itself (plus a Redis instance it depends on) — it does not run the data engine. Copy agent_deployment/Docker/env_sample to agent_deployment/Docker/.env and fill in your AWS credentials before running docker compose up in that directory.
• agent_deployment/Full-install/ and agent_deployment/Using-AMI/ contain the CloudFormation templates (deploy-agent.yml, deploy-agent-AWS-AMI.yml) that the Flask app submits — these boot a plain EC2 instance (or one from a pre-baked AMI) and run the same "Launch the Agentic Interface" steps above via cloud-init.

If you're looking to containerize the agent itself for local testing, that setup doesn't exist yet in this repo.

Talk to Task: Example Run

User: Hi, I would like to explore the data engine for processing my data.

Agent: Great! I can help you with Auto Labeling — training or running an object
       detection model on your dataset. Which dataset would you like to use?

User: fisheye8k

Agent: Dataset 'fisheye8k' confirmed. Do you have credentials configured for
       CVAT or Label Studio? I detected Label Studio credentials — using
       Label Studio for annotation. How would you like to annotate your
       dataset?
          - Manual Labeling: export your dataset for manual annotation.
          - Auto Generated Labeling: run a detection model to generate
            predictions automatically, then review/correct them.

User: Auto Generated Labeling.

Agent: Here are the available model sources and models: [...]
       Which model would you like to use?

User: rfdetr_2xlarge

Agent: rfdetr_2xlarge (roboflow) has been configured.
       Here are the current hyperparameters: [...]
       Would you like to modify any of these hyperparameters, or are you
       ready to start?

User: Let's go with the defaults.

Agent: Here's a summary of what will be run: [...] Shall I proceed with
       auto-labeling?

User: Yes, proceed.

[Data Engine Server]: Running auto-labeling — live logs are streamed here and to Weights and Biases.

Agent: Auto-labeling completed. Predictions have been exported to Label Studio
       for review. Let me know when you're done and I'll import your labels
       back, or I can launch Voxel51 so you can explore the results now.

Notebooks and Submodules

To exclude the output of jupyter notebooks from git tracking, add the following lines to your .git/config :

[filter "strip-notebook-output-engine"]
    clean = <your_path>/mcity_data_agent/.venv/bin/jupyter nbconvert --ClearOutputPreprocessor.enabled=True --ClearMetadataPreprocessor.enabled=True --to=notebook --stdin --stdout
    smudge = cat
    required = true

and those to .git/modules/mcity_data_engine_scripts/config

[filter "strip-notebook-output-scripts"]
    clean = <your_path>/mcity_data_agent/.venv/bin/jupyter nbconvert --ClearOutputPreprocessor.enabled=True --ClearMetadataPreprocessor.enabled=True --to=notebook --stdin --stdout
    smudge = cat
    required = true

In order to keep the submodules updated, add the following lines to the top of your .git/hooks/pre-commit:

git submodule update --recursive --remote
git add .gitmodules $(git submodule foreach --quiet 'echo $name')

Repository Structure

.
├── main.py                     # Entry point of the framework → Terminal 1
├── session_v51.py              # Script to launch Voxel51 session → Terminal 2
├── workflows/                  # Workflows for the Mcity Data Engine
├── config/                     # Local configuration files
├── utils/                      # General-purpose utility functions
├── cloud/                      # Scripts run in the cloud to pre-process data
├── docs/                       # Documentation generated with `pdoc`
├── tests/                      # Tests using Pytest
├── custom_models/              # External models with containerized environments
├── mcp_layer/                   # Agentic (MCP) layer
│  ├── mcp_server.py            # MCP tool registry (port 8000)
│  ├── chat_server.py           # FastAPI chat endpoint (port 8001)
│  ├── ingest_server.py         # File upload & processing (port 8002)
│  ├── client_chat.py           # Web/terminal client (port 5225)
│  ├── chat_pipeline.py         # Tool dispatch, state routing, reply formatting
│  ├── validate_workflow_state.py # Pydantic workflow state machine
│  ├── mcptools/                # Tool implementations
│  │   ├── __init__.py
│  │   ├── workflow_selector.py
│  │   ├── auto_labeling.py
│  │   ├── data_ingest.py
│  │   ├── cvat_export.py
│  │   ├── label_studio_export.py
│  │   └── v51.py               # Voxel51 integration
│  ├── llm_clients.py           # Multi-LLM support (OpenAI, Claude, Gemini, Groq)
│  ├── tool_schema.py           # Tool definitions exposed to the LLM
│  ├── prompts/                 # System + per-workflow prompt text
│  │   └── workflows/
│  └── ui/                      # Web interface assets
│      └── index.html
├── agent_deployment/            # AWS CloudFormation / Docker launcher for the agent
├── mcity_data_engine_scripts/   # Experiment scripts and one-time operations (Mcity internal)
├── .vscode                     # Settings for VS Code IDE
├── .github/workflows/          # GitHub Action workflows
├── .gitignore                  # Files and directories to be ignored by Git
├── .gitattributes              # Rules for handling files like Notebooks during commits
├── .gitmodules                 # Configuration for managing Git submodules
├── .secret                     # Secret tokens (not tracked by Git)
└── requirements.txt            # Python dependencies (pip install -r requirements.txt)

Training

Training runs are logged with Weights and Biases (WandB).

In order to change the standard WandB directory, run

echo 'export WANDB_DIR="<your_path>/mcity_data_agent/logs"' >> ~/.profile
source ~/.profile

Contribution

Contributions are very welcome! The Mcity Data Engine is a blueprint for data curation and model training and will not support every use case out of the box. Please find instructions on how to contribute here:

• Contribute new workflow
• Contribute new dataset
• Contribute new model

Special thanks to these amazing people for contributing to the Mcity Data Engine! 🙌

Citation

If you use the Mcity Data Engine in your research, feel free to cite the project:

@article{bogdoll2025mcitydataengine,
  title={Mcity Data Engine},
  author={Bogdoll, Daniel and Anata, Rajanikant Patnaik and Stevens, Gregory},
  journal={GitHub. Note: https://github.com/mcity/mcity_data_engine},
  year={2025}
}