Escalator

Give it a URL, get clean Markdown. It climbs the cheapest rung that works — a plain HTTP fetch, the same fetch through a residential proxy, then a stealth browser — and stops at the first one that comes back with real content.

$ escalator scrape https://en.wikipedia.org/wiki/Web_scraping | head -3
# Web scraping

**Web scraping**, **web harvesting**, or **web data extraction** is [data scraping](...)

This README is an installation manual. It walks the whole way from a bare machine to a running, authenticated HTTP API on three targets — Linux, macOS, and Docker — with nothing left to guess. If you only want the one-page tour of why it is shaped this way, that lives in DESIGN.md.

Contents

• What you are installing
• Install on Linux
  • 1. System libraries the browser needs
  • 2. Install Python and escalator
  • 3. Give it a browser
  • 4. Configure it
  • 5. Check the install
  • 6. Run it as an API
• Install on macOS
• Install with Docker
• Configuration reference
• Using the API
• Command reference
• The ladder
• Troubleshooting
• Development · License

What you are installing

escalator is a single Python package (escalator on PyPI) that is three things in one binary:

• a CLI — escalator scrape URL prints Markdown to stdout, so it pipes;
• an HTTP API — escalator serve exposes POST /scrape, bearer-authenticated;
• an MCP endpoint — the same ladder as one tool at /mcp, for agents (the escalator[mcp] extra).

The pieces you need to have present:

It writes nothing outside its own config and data directories, ships no telemetry, and bundles no browser or proxies.

Install on Linux

The instructions below are distro-neutral. Pick the package-manager line that matches yours; everything after step 1 is identical.

1. System libraries the browser needs

On a desktop these are already present — skip to step 2. On a slim server or container Chrome will resolve but refuse to launch with error while loading shared libraries: libnss3.so until you install them.

Debian / Ubuntu (apt):

sudo apt-get update && sudo apt-get install -y \
  libnss3 libnspr4 libatk1.0-0t64 libatk-bridge2.0-0t64 libcups2t64 libdrm2 \
  libxkbcommon0 libxcomposite1 libxdamage1 libxext6 libxfixes3 libxrandr2 \
  libgbm1 libglib2.0-0t64 libpango-1.0-0 libcairo2 libasound2t64 \
  libatspi2.0-0t64 libxcb1 libdbus-1-3 libexpat1

The t64 suffixes are Ubuntu 24.04 / Debian 13. On older releases apt will tell you the names it wants (libatk1.0-0, libcups2, libasound2, …) — drop the t64 and re-run.

Fedora / RHEL / CentOS Stream / Rocky / Alma (dnf):

sudo dnf install -y \
  nss nspr atk at-spi2-atk at-spi2-core cups-libs libdrm \
  libxkbcommon libXcomposite libXdamage libXext libXfixes libXrandr \
  mesa-libgbm glib2 pango cairo alsa-lib libxcb dbus-libs expat

Arch / Manjaro (pacman):

sudo pacman -S --needed \
  nss nspr atk at-spi2-atk at-spi2-core cups libdrm \
  libxkbcommon libxcomposite libxdamage libxext libxfixes libxrandr \
  mesa glib2 pango cairo alsa-lib libxcb dbus expat

Alpine (musl libc): Chrome for Testing is a glibc build and will not run here. Install the distro's chromium instead and point escalator at it:

sudo apk add chromium nss freetype harfbuzz ttf-freefont
export ESCALATOR_BROWSER_PATH=/usr/bin/chromium-browser

Shortcut for any distro: installing your distro's own chromium (or google-chrome) package pulls in every one of these libraries as a dependency. You can install it purely to satisfy the libs and still let escalator drive its own pinned Chrome — or just use that browser directly.

2. Install Python and escalator

Use uv — it installs the tool onto your PATH and provisions a correct Python (3.11–3.13) even if the system one is too new or too old.

# install uv itself (once)
curl -LsSf https://astral.sh/uv/install.sh | sh
exec $SHELL              # pick up the new PATH

# install escalator, with the MCP endpoint included
uv tool install "escalator[mcp]"

That puts an escalator binary in ~/.local/bin. Confirm PATH is set:

escalator --version
# escalator 0.1.0 | python 3.13.x | Linux-x86_64

# pipx — same idea as uv tool
pipx install "escalator[mcp]"

# or an explicit virtualenv you manage yourself
python3 -m venv ~/.venvs/escalator
~/.venvs/escalator/bin/pip install "escalator[mcp]"
~/.venvs/escalator/bin/escalator --version

3. Give it a browser

escalator never downloads software as a side effect of a scrape. Provision the browser once, explicitly. Either is fine:

# A) download a pinned Chrome for Testing into escalator's own data dir
escalator browser install

# B) or just rely on a Chrome/Chromium/Edge/Brave already installed — check:
escalator browser list      # shows every candidate and marks the winner (→)

4. Configure it

Two supported styles — use whichever fits how you'll run it:

Interactive, writes ~/.config/escalator/config.toml:

escalator init              # add --yes to take every default non-interactively

Environment only (no file) — this is how the API service below is configured. Every setting has an ESCALATOR_* variable; see the configuration reference.

5. Check the install

escalator doctor

doctor checks Python, the config file, the data directory, browser resolution, an actual headless launch (this is what catches missing libs), and — if a proxy is set — one real request through it, reporting the egress IP and country with the password masked. Every ❌ prints the one line that fixes it, and the exit code is non-zero on any failure, so scripts can gate on it. Add --json for machines.

You are done with the CLI. To expose it as a network service, continue.

6. Run it as an API (Linux)

The goal: escalator running as an unprivileged system service, bound to loopback, with an nginx reverse proxy terminating TLS in front of it.

6a. Create a dedicated user, venv, and data directory

Isolating the service from your login account keeps its data, browser, and crash blast-radius contained.

# a locked-down system user that owns the install and its data
sudo useradd --system --home-dir /opt/escalator --shell /usr/sbin/nologin escalator
sudo install -d -o escalator -g escalator /opt/escalator /var/lib/escalator

# install escalator into a venv owned by that user
sudo -u escalator python3 -m venv /opt/escalator/venv
sudo -u escalator /opt/escalator/venv/bin/pip install --upgrade pip
sudo -u escalator /opt/escalator/venv/bin/pip install "escalator[mcp]"

# give the service its own browser, inside its data dir
sudo -u escalator env ESCALATOR_STORAGE_DATA_DIR=/var/lib/escalator \
  /opt/escalator/venv/bin/escalator browser install

6b. Mint an API key and write the environment file

The API rejects every request unless at least one bearer key is configured. The server.api_keys list is the source of truth — removing a key revokes it on the next restart. Mint one with the built-in command; --show prints a fresh key to stdout without writing any file, which is exactly what an env-based deploy wants:

KEY=$(sudo -u escalator /opt/escalator/venv/bin/escalator key generate --show)
echo "$KEY"     # save this — it is the credential

Put configuration in an env file the service reads (root-owned, mode 600 — it holds the key and any proxy password):

sudo install -d -m 755 /etc/escalator
sudo tee /etc/escalator/escalator.env >/dev/null <<EOF
# --- required ---
ESCALATOR_SERVER_API_KEYS=$KEY
# Bind to loopback: nginx (below) is the only thing that should reach it.
ESCALATOR_SERVER_HOST=127.0.0.1
ESCALATOR_SERVER_PORT=8000
ESCALATOR_STORAGE_DATA_DIR=/var/lib/escalator

# --- optional: a residential proxy for the middle rungs ---
# ESCALATOR_PROXY_ENABLED=true
# ESCALATOR_PROXY_URL=http://user:[email protected]:8080
# several exits, round-robin:
# ESCALATOR_PROXY_LIST=http://user:[email protected]:8080,http://user:[email protected]:8080

# --- optional: politeness ---
ESCALATOR_POLITENESS_RESPECT_ROBOTS=true
ESCALATOR_POLITENESS_RATE_LIMIT_RPS=1.0
EOF
sudo chmod 600 /etc/escalator/escalator.env

Multiple keys are comma-separated: ESCALATOR_SERVER_API_KEYS=key1,key2.

6c. Supervise the process

Use systemd (native to every modern Linux) or Supervisor — pick one.

systemd — write /etc/systemd/system/escalator.service:

[Unit]
Description=escalator scraping gateway
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=escalator
Group=escalator
EnvironmentFile=/etc/escalator/escalator.env
ExecStart=/opt/escalator/venv/bin/escalator serve
Restart=on-failure
RestartSec=3

# hardening — the service only needs to write its own data dir
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/escalator

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now escalator
sudo systemctl status escalator
journalctl -u escalator -f          # follow the logs
curl -s localhost:8000/healthz      # {"status":"ok","version":"0.1.0"}

Supervisor — as an alternative, drop this into /etc/supervisor/conf.d/escalator.conf:

[program:escalator]
command=/opt/escalator/venv/bin/escalator serve
user=escalator
directory=/var/lib/escalator
autostart=true
autorestart=true
startsecs=5
stopasgroup=true
killasgroup=true
; Supervisor splits `environment` on commas, so a value that itself contains
; commas (several API keys, a proxy list) MUST be quoted as one item:
environment=ESCALATOR_SERVER_API_KEYS="PASTE_KEY_HERE",ESCALATOR_SERVER_HOST="127.0.0.1",ESCALATOR_SERVER_PORT="8000",ESCALATOR_STORAGE_DATA_DIR="/var/lib/escalator"
stdout_logfile=/var/log/escalator.log
redirect_stderr=true

sudo supervisorctl reread && sudo supervisorctl update
sudo supervisorctl status escalator

6d. Put a web server in front of it (TLS)

escalator serves plain HTTP on loopback and runs as a single process — it is not meant to face the internet directly. Terminate TLS and expose it with nginx. Write /etc/nginx/sites-available/escalator:

server {
    listen 80;
    server_name scrape.example.com;
    # redirect everything to HTTPS (certbot will fill in the 443 block)
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    http2 on;
    server_name scrape.example.com;

    ssl_certificate     /etc/letsencrypt/live/scrape.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/scrape.example.com/privkey.pem;

    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # A browser climb can take 30s+. Don't let nginx hang up mid-scrape.
        proxy_read_timeout 120s;
        proxy_connect_timeout 10s;
    }
}

sudo ln -s /etc/nginx/sites-available/escalator /etc/nginx/sites-enabled/
sudo nginx -t                                   # check config
sudo certbot --nginx -d scrape.example.com      # obtain + wire up a cert
sudo systemctl reload nginx

Test end to end:

curl -s https://scrape.example.com/healthz
curl -s https://scrape.example.com/scrape \
  -H "Authorization: Bearer PASTE_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}' | head

Prefer Caddy? A whole Caddyfile for automatic TLS is just:

scrape.example.com {
    reverse_proxy 127.0.0.1:8000
}

Install on macOS

For desktop use and local development. The system libraries step does not apply — macOS already has what the browser needs.

1. Prerequisites

# Homebrew, if you don't have it: https://brew.sh
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

# uv, to install the tool and manage its Python
brew install uv

A Chrome-family browser is usually already installed. If not, either brew install --cask google-chrome, or let escalator fetch its own in step 3.

2. Install escalator

uv tool install "escalator[mcp]"
escalator --version
# escalator 0.1.0 | python 3.13.x | Darwin-arm64

3. Browser, config, and check

escalator browser list       # is Chrome/Chromium/Edge/Brave already here?
escalator browser install    # …or download a pinned one into the data dir

escalator init               # write ~/Library/Application Support/escalator/config.toml
escalator doctor             # verify everything, including a real headless launch

You can now escalator scrape https://example.com. To run the API continuously on the Mac, continue.

4. Run it as an API (macOS)

4a. Configuration

Mint a key straight into the init config file — no hand-editing:

escalator key generate
# → prints the key and appends it to ~/Library/Application Support/escalator/config.toml

That writes api_keys = ["…"] under [server]. While you're there you can set host and port too, though 127.0.0.1:8000 is already the default.

(Or skip the file and pass everything as ESCALATOR_* env vars in the plist below — the plist wins over the file either way. escalator key generate --show prints a fresh key for that without touching the config.)

4b. Keep it running with launchd

launchd is the native macOS supervisor. Create ~/Library/LaunchAgents/com.escalator.serve.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.escalator.serve</string>

    <key>ProgramArguments</key>
    <array>
        <string>/Users/YOU/.local/bin/escalator</string>
        <string>serve</string>
    </array>

    <key>EnvironmentVariables</key>
    <dict>
        <key>ESCALATOR_SERVER_API_KEYS</key>
        <string>PASTE_A_GENERATED_KEY</string>
        <key>ESCALATOR_SERVER_HOST</key>
        <string>127.0.0.1</string>
        <key>ESCALATOR_SERVER_PORT</key>
        <string>8000</string>
    </dict>

    <key>RunAtLoad</key>   <true/>
    <key>KeepAlive</key>   <true/>
    <key>StandardOutPath</key>  <string>/tmp/escalator.log</string>
    <key>StandardErrorPath</key><string>/tmp/escalator.err</string>
</dict>
</plist>

Replace /Users/YOU with your home (echo $HOME) and confirm the binary path with which escalator. Then:

launchctl load ~/Library/LaunchAgents/com.escalator.serve.plist
curl -s localhost:8000/healthz
# to stop: launchctl unload ~/Library/LaunchAgents/com.escalator.serve.plist

4c. A web server in front (optional)

If this Mac serves other machines, put nginx in front for TLS exactly as in the Linux nginx step:

brew install nginx        # config lives at /opt/homebrew/etc/nginx/

Use the same server { … } block; on a Mac you'll typically manage certs with brew install certbot or point at certs you already have.

Install with Docker

The published image (ghcr.io/ruslanstarikov/escalator) carries a pinned Chrome and the MCP extra, runs as an unprivileged user, and is configured entirely by environment variable — no config file is ever mounted. This is the least fiddly way to run the API on a server.

One-shot commands

The default command prints help; give it a subcommand to do work.

# check the image on this host (browser launch, libs, everything)
docker run --rm ghcr.io/ruslanstarikov/escalator doctor

# scrape a single URL to stdout
docker run --rm ghcr.io/ruslanstarikov/escalator scrape https://example.com

# mint a bearer key to paste into the env below (--show writes no file)
docker run --rm ghcr.io/ruslanstarikov/escalator key generate --show

Run the API with docker run

docker run -d --name escalator \
  -e ESCALATOR_SERVER_API_KEYS=PASTE_A_GENERATED_KEY \
  -p 127.0.0.1:8000:8000 \
  -v escalator-data:/data \
  --shm-size=512m \
  --restart unless-stopped \
  ghcr.io/ruslanstarikov/escalator serve

What each flag is doing, so nothing is mysterious:

The image already sets ESCALATOR_SERVER_HOST=0.0.0.0, ESCALATOR_STORAGE_DATA_DIR=/data, and ESCALATOR_BROWSER_PATH to the baked-in browser, so you don't repeat those.

docker logs -f escalator
curl -s localhost:8000/healthz

Run the API with Docker Compose (recommended)

Create a .env next to the compose file (never commit it):

ESCALATOR_SERVER_API_KEYS=PASTE_A_GENERATED_KEY
# optional residential proxy:
# ESCALATOR_PROXY_ENABLED=true
# ESCALATOR_PROXY_URL=http://user:[email protected]:8080

docker-compose.yml:

services:
  escalator:
    image: ghcr.io/ruslanstarikov/escalator:latest
    command: ["serve"]
    environment:
      ESCALATOR_SERVER_API_KEYS: "${ESCALATOR_SERVER_API_KEYS:?set a key in .env}"
      ESCALATOR_PROXY_ENABLED:   "${ESCALATOR_PROXY_ENABLED:-false}"
      ESCALATOR_PROXY_URL:       "${ESCALATOR_PROXY_URL:-}"
      ESCALATOR_BROWSER_VIA_PROXY: "true"
      ESCALATOR_POLITENESS_RESPECT_ROBOTS: "true"
      ESCALATOR_POLITENESS_RATE_LIMIT_RPS: "1.0"
    ports:
      - "127.0.0.1:8000:8000"   # loopback; put a reverse proxy in front for TLS
    volumes:
      - ./data:/data            # must be writable by uid 1000 (see note)
    restart: unless-stopped
    shm_size: 512mb

docker compose up -d
docker compose logs -f
curl -s localhost:8000/healthz

A ready-to-copy version with the full proxy wiring is in docker-compose.example.yml.

Volume permissions. The container runs as uid 1000. Docker Desktop (macOS/Windows) maps this automatically. On a Linux host, a bind-mounted ./data must be writable by that uid — either sudo chown -R 1000:1000 ./data once, or add user: "$(id -u):$(id -g)" to the service. A named volume (escalator-data:/data) sidesteps this entirely.

TLS in front of the container

Terminate TLS with the same nginx block from the Linux section pointing at http://127.0.0.1:8000, or add a Caddy/Traefik service to the compose file. Because the container publishes only to 127.0.0.1, the proxy is the sole path in from the network.

Build the image yourself

git clone https://github.com/ruslanstarikov/escalator.git
cd escalator
docker build -t escalator:local .
docker run --rm escalator:local doctor

The build pins its Chrome version via a --build-arg CHROME_VERSION=…; see the comments at the top of the Dockerfile. On arm64 (where Chrome for Testing has no build) it installs the distro's chromium instead, so the same ESCALATOR_BROWSER_PATH works on both architectures.

Configuration reference

One file, written by init, at the platform config dir (~/.config/escalator/config.toml on Linux, ~/Library/Application Support/escalator/config.toml on macOS). Override the location with --config.

Precedence, everywhere:

CLI flag  >  environment  >  config.toml  >  default

Every key has an environment variable — which is how the Docker image and the system service are configured without a file at all:

Data — the SQLite database and any downloaded browser — lives in the platform data dir, overridable with ESCALATOR_STORAGE_DATA_DIR. Nothing is ever written outside it.

Where the browser comes from

escalator browser list shows the search, in order:

1. an explicit path — --browser-path, then ESCALATOR_BROWSER_PATH, then browser.path. If set and wrong, that's an error naming the path, never a silent fall-through.
2. browsers installed on this machine: real Google Chrome first, then Chromium, then Edge and Brave.
3. a browser escalator browser install downloaded earlier.

If none of those find anything, you get an error naming the two commands that fix it. Resolution never downloads on its own — a server request or a cron job should not install software as a side effect.

Using the API

escalator serve   # binds ESCALATOR_SERVER_HOST:PORT (127.0.0.1:8000 by default)

POST /scrape   {url, markdown?, min_tier?, max_tier?, timeout_ms?}  -> FetchResult
GET  /healthz                                                       -> {status, version}

Authenticate every /scrape call with Authorization: Bearer <key>, where the key is one of server.api_keys. That list is the truth: remove a key and it is revoked on the next start. Mint one with escalator key generate; there is deliberately no HTTP endpoint to do so — a running server should not be able to grant itself credentials. /healthz needs no auth — it's what a load balancer polls.

curl -s http://127.0.0.1:8000/scrape \
  -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com"}'

A wall comes back as 200 OK with {"status": "challenged"}, not an HTTP error — deliberately, so an agent on the other end can react. Retrying in a loop won't help: escalator does not solve CAPTCHAs, by design. A residential proxy is the usual answer.

With the [mcp] extra the same ladder is exposed at /mcp as one tool, scrape_url(url, force_browser=False), for MCP-speaking agents.

Command reference

escalator init [--yes]     configure this machine; --yes for scripts
escalator doctor [--json]  check everything, one fix per failure
escalator browser list     every browser found, and which one wins
escalator browser install  download Chrome for Testing into the data dir
escalator key generate     mint a bearer key into the config (--show: print only)
escalator scrape URL       one page to stdout, so it pipes (--html for raw HTML)
escalator serve            the HTTP API and the MCP face
escalator --version        tool, python, platform

The ladder

policy      robots.txt (cached) + rate limit  → may short-circuit (skip/deny/wait)
http        curl_cffi, impersonate=chrome     → ~100ms; clears undefended sites
http_proxy  same, routed via residential IP   → beats datacenter-IP bans
browser     nodriver, headless Chrome         → JS/SPA + Cloudflare-class defenses
                    │
                    └─ walled on the last rung? → status="challenged". Surrender.

Two things make this more than a for loop:

200 OK is not success. A rung that returns HTTP 200 carrying a Cloudflare interstitial has not succeeded. core/detect.py classifies every response after extraction — content, thin, or blocked — and only content counts. Without that the ladder would never escalate, and the cache would learn "http works" for a domain that serves junk forever.

The cache forgets. A learned start-rung that only ratcheted upward would drift every domain toward browser+proxy and quietly inflate your proxy bill. Entries carry learned_at; past ladder.tier_cache_ttl_hours a domain retries one rung cheaper.

See DESIGN.md for why it is shaped this way — and for what it deliberately refuses to do.

Troubleshooting

Start here — always:

escalator doctor

Every ❌ comes with the one line that fixes it. Common cases:

If it still won't work, paste the whole escalator doctor output into an issue — that's what its last line asks for, and the fastest route to an answer.

Development

See CONTRIBUTING.md. In short: uv sync, uv run pytest.

License

Released into the public domain — see UNLICENSE. No warranty, no attribution required, do what you like with it.