Lockwood Group Observability

Semantic log clustering for incident triage, exposed as an MCP server.

What it does and why

During an incident, the log stream is mostly repetition: the same failure printed thousands of times with different ids, plus a few distinct signals buried in the noise. This server takes a window of log lines and groups them by semantic similarity, then ranks the groups so the ones that matter for triage surface first. Instead of scrolling raw logs, an operator (or an agent driving the MCP tool) sees "3 clusters: a CRITICAL checkout 500 seen 240x, an ERROR payment timeout seen 1900x, ...".

Triage runs on the incident path, so latency is a hard requirement, not a nice-to-have: a triage_logs call has a p99 budget under 150ms. That budget shapes the whole design (see Architecture and docs/adr/).

Architecture

inkwellobservability/
  types.py                 domain types: LogRecord, Cluster, TriageResult, Severity
  clustering.py            TriageEngine + normalize_message (the core algorithm)
  config.py                env-driven Config with startup validation
  errors.py                ValidationError / ResourceLimitError / ConfigError
  logging_setup.py         JSON-lines logging to stderr
  service.py               argument validation + engine orchestration
  server.py                MCP JSON-RPC 2.0 server over stdio
  __main__.py              entry point (python -m inkwellobservability)
  bench.py                 latency benchmark (make bench)
  providers/
    base.py                EmbeddingProvider interface
    stub.py                deterministic offline embeddings (hot path)
    real.py                HTTP embedding client (optional, never used in tests)
tests/
docs/adr/                  architecture decision records

The request path: server parses a JSON-RPC line and dispatches tools/call to service, which validates arguments, enforces resource guards, and hands LogRecords to the TriageEngine. The engine normalizes each message to a template, embeds it via the configured EmbeddingProvider, and assigns it to the nearest cluster in a single greedy pass.

Two mechanisms make the 150ms budget a property of the design rather than a hope:

• Bounded cluster count K. Assignment scans existing centroids, so per-record cost is O(K * dim). Once K clusters exist, further distinct log shapes fold into the nearest cluster instead of spawning new scans. Total cost is O(N * K * dim) with a known constant.
• A wall-clock deadline. The clustering loop checks elapsed time on a coarse stride and, once the budget is spent, folds the remaining records into existing clusters and flags the result degraded — a partial but usable ranking, returned inside the budget, rather than running long.

Embeddings on the hot path come from a deterministic feature-hashing provider (StubEmbeddingProvider) with no network hop. RealEmbeddingProvider is an optional quality upgrade behind the same interface; it is never required for the tests, which run fully offline. The reasoning behind these calls is recorded in docs/adr/.

Install

make venv       # python3 -m venv .venv
make install    # .venv/bin/python -m pip install -e .[dev]

All Makefile targets use $(PY) (default .venv/bin/python); override it to use a different interpreter, e.g. make test PY=python3.

Quickstart

Run the test suite and the latency benchmark:

make test PY=python3
make bench PY=python3

make bench prints, for a 2000-record window:

records/window : 2000
budget         : 150.0 ms
p50            : ...
p99            : ...
within budget  : True

Use the engine directly as a library:

from inkwellobservability import TriageEngine, LogRecord

engine = TriageEngine()  # defaults: threshold 0.55, K=64, budget 150ms
records = [
    LogRecord.of("payment gateway timeout for order 4821", level="ERROR"),
    LogRecord.of("payment gateway timeout for order 90", level="ERROR"),
    LogRecord.of("user 12 login failed", level="WARNING"),
]
result = engine.triage(records)
for cluster in result.clusters:            # ranked, worst first
    print(cluster.size, cluster.max_level.name, cluster.template)
print("degraded:", result.degraded, "elapsed_ms:", result.elapsed_ms)

Ranking weights frequency by worst severity seen (size * 4**level): a small burst of CRITICAL out-ranks a flood of INFO, while a large ERROR cluster still out-ranks a single CRITICAL. See docs/adr/0004-severity-weighted-ranking.md.

As an MCP server

The entry point speaks MCP (JSON-RPC 2.0) over stdio, one JSON object per line:

python -m inkwellobservability          # serve on stdin/stdout
python -m inkwellobservability --version
python -m inkwellobservability --help

After make install the same server is on PATH as inkwell-observability.

It exposes one tool, triage_logs, taking either inline records or a path to a JSON-lines file, plus an optional budget_ms. Piping two requests in:

printf '%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize"}' \
  '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"triage_logs","arguments":{"records":[{"message":"payment timeout order 1","level":"ERROR"},{"message":"payment timeout order 2","level":"ERROR"}]}}}' \
  | python -m inkwellobservability

The tool result carries both a text block (JSON) and structuredContent with clusters (ranked worst-first), total_records, cluster_count, elapsed_ms, budget_ms, and degraded.

Logs are JSON lines on stderr (stdout is reserved for protocol messages). Every triage call logs its timing; a call that spends its budget logs at WARNING with degraded: true.

Configuration reference

All configuration is environment-driven and validated at startup; an invalid value exits non-zero before the server starts serving.

Variable	Default	Meaning
INKWELL_PROVIDER	stub	stub (offline) or real (HTTP embeddings)
INKWELL_EMBEDDING_DIM	128	embedding dimension
INKWELL_SIMILARITY_THRESHOLD	0.55	cosine threshold for joining a cluster
INKWELL_MAX_CLUSTERS	64	cluster cap K (bounds per-record cost)
INKWELL_BUDGET_MS	150.0	per-call latency budget
INKWELL_MAX_RECORDS	20000	resource guard: max records per call
INKWELL_MAX_MESSAGE_BYTES	16384	resource guard: max bytes per message
INKWELL_LOG_LEVEL	INFO	DEBUG/INFO/WARNING/ERROR

When INKWELL_PROVIDER=real, the real provider also reads INKWELL_EMBED_ENDPOINT and INKWELL_EMBED_API_KEY. If either is missing (or the backend is unreachable at construction), the service logs a warning and degrades to the stub provider rather than failing to start.

Failure handling

• Malformed JSON-RPC line, bad envelope, or unknown method -> JSON-RPC error.
• Bad tool arguments, missing/unreadable file, malformed JSONL line, or a resource guard trip -> a tools/call result with isError: true and a message; the server loop stays up.
• Oversized batch or message is rejected up front so one request cannot blow the latency budget for others.

Known limitations

• Per-call windows, no streaming state. Each triage_logs call clusters the window it is given; clusters are not carried across calls. A rolling window with centroid eviction is deferred until the ingestion shape is fixed (TODO in clustering.py). See docs/adr/0001.
• Feature-hashing embeddings are shallow. The default provider matches on shared tokens, not learned meaning, so paraphrases with no common words ("disk full" vs "no space left on device") will not cluster together. Point INKWELL_PROVIDER=real at an embedding backend for semantic recall, at the cost of the offline guarantee. See docs/adr/0002.
• Greedy clustering is order-sensitive. Assignment depends on arrival order; a different permutation can yield slightly different centroids. This is an accepted tradeoff for the single-pass cost bound (docs/adr/0001).
• Single tool, no auth. The MCP surface is triage_logs only, and the stdio transport assumes a trusted local client. Network transport and authentication are out of scope for this deliverable.

---

Lockwood Group is an illustrative client; this repository is a self-directed reference implementation built to work end to end.