Usa Open Data Connectors

TypeScript connectors for US public data, with language-agnostic wrappers so you can use them from any language (Python, R, Julia, curl, whatever you like).

Keyless-first: every connector works without an API key. Keys, when a source needs one, are read from the environment and stay server-side. They are never exposed over the API or committed to the repo.

Packages

Package	What it is
@usa-open-data-connectors/usa-sources	Uniform adapters for US public data sources, with live probes and offline fixtures
@usa-open-data-connectors/usa-mcp	MCP server that exposes the connectors to Claude, ChatGPT, and other MCP clients
@usa-open-data-connectors/connectors-cli	usdata command line tool that prints JSON to stdout, so any language can shell out to it
@usa-open-data-connectors/connectors-api	Private. HTTP wrapper with an OpenAPI spec and Swagger UI, so any language can call the connectors over HTTP
@usa-open-data-connectors/config-eslint	Private. Shared ESLint flat configuration
@usa-open-data-connectors/config-typescript	Private. Shared TypeScript compiler settings

Connectors

Adapters, all in @usa-open-data-connectors/usa-sources. All are keyless and need no environment variable.

Most of this data comes from US federal agencies, and works of the US federal government are generally not subject to copyright in the United States, so it is effectively public domain there. EstimateTax is a private publisher, and its terms ask for attribution and the source document alongside any figure you display. The NCBI and NLM services attach their own terms to some content, and public domain in the US is not public domain everywhere. Check the publisher's terms before you republish: the connectors fetch the data, they do not relicense it. docs/CONNECTOR_DISCOVERY.md lists where each source's terms live. Socrata and CKAN catalogue entries carry the portal's licenceUrl and attribution values, and PublicDataHub records carry licence and agency provenance. An empty value means the source supplied none.

The SEC EDGAR, National Weather Service and CFPB adapters ask callers to identify themselves. They send their own descriptive User-Agent header from inside the adapter, which replaces the default one, so callers send nothing extra. Other adapters send the shared User-Agent (usa-open-data-connectors (Language=TypeScript)), waits at most 30 seconds for a response, and marks rate-limited (HTTP 429), server-error (HTTP 5xx) and network failures as retryable on the thrown UsSourceApiError. The shared HTTP helpers (httpGet and httpPostJson) in packages/usa-sources/src/http.ts do this for every adapter.

id	Source	Keyless?	Example command
bls-unemployment-rate	Bureau of Labor Statistics, national unemployment rate	Yes	npx tsx packages/cli/src/cli.ts probe bls-unemployment-rate
usgs-hawaii-earthquakes	US Geological Survey, earthquakes near Hawaii	Yes	npx tsx packages/cli/src/cli.ts probe usgs-hawaii-earthquakes
cdc-county-obesity	CDC PLACES county obesity estimates	Yes	npx tsx packages/cli/src/cli.ts probe cdc-county-obesity
ncei-annual-temperature	NOAA NCEI contiguous US annual average temperature	Yes	npx tsx packages/cli/src/cli.ts probe ncei-annual-temperature
noaa-sea-level	NOAA CO-OPS monthly mean sea level	Yes	npx tsx packages/cli/src/cli.ts probe noaa-sea-level
treasury-avg-interest-rate	US Treasury average interest rate on the debt	Yes	npx tsx packages/cli/src/cli.ts probe treasury-avg-interest-rate
fema-disaster-declarations	FEMA disaster declarations	Yes	npx tsx packages/cli/src/cli.ts probe fema-disaster-declarations
openfda-food-recalls	openFDA food enforcement reports	Yes	npx tsx packages/cli/src/cli.ts probe openfda-food-recalls
usgs-peak-streamflow	USGS annual peak streamflow at St. Louis	Yes	npx tsx packages/cli/src/cli.ts probe usgs-peak-streamflow
cpsc-product-recalls	CPSC consumer product recalls	Yes	npx tsx packages/cli/src/cli.ts probe cpsc-product-recalls
cfpb-consumer-complaints	CFPB Consumer Complaint Database	Yes	npx tsx packages/cli/src/cli.ts probe cfpb-consumer-complaints
treasury-debt-to-penny	US Treasury debt to the penny	Yes	npx tsx packages/cli/src/cli.ts probe treasury-debt-to-penny
sec-edgar-filings	SEC EDGAR issuer submissions and recent filings	Yes	npx tsx packages/cli/src/cli.ts probe sec-edgar-filings
fdic-bank-directory	FDIC bank directory	Yes	npx tsx packages/cli/src/cli.ts probe fdic-bank-directory
clinicaltrials-studies	ClinicalTrials.gov registered studies	Yes	npx tsx packages/cli/src/cli.ts probe clinicaltrials-studies
nws-point-forecast	National Weather Service forecast grid point	Yes	npx tsx packages/cli/src/cli.ts probe nws-point-forecast
usaspending-agencies	USAspending top-tier federal agencies	Yes	npx tsx packages/cli/src/cli.ts probe usaspending-agencies
epa-envirofacts-facilities	EPA Envirofacts Toxics Release Inventory facilities	Yes	npx tsx packages/cli/src/cli.ts probe epa-envirofacts-facilities
ncbi-pubmed-search	NCBI PubMed literature search	Yes	npx tsx packages/cli/src/cli.ts probe ncbi-pubmed-search
nih-reporter-projects	NIH RePORTER funded projects	Yes	npx tsx packages/cli/src/cli.ts probe nih-reporter-projects
cdc-socrata-catalogue	CDC Socrata catalogue	Yes	npx tsx packages/cli/src/cli.ts probe cdc-socrata-catalogue
healthdata-socrata-catalogue	HealthData.gov Socrata catalogue	Yes	npx tsx packages/cli/src/cli.ts probe healthdata-socrata-catalogue
bts-transportation-stats	Bureau of Transportation Statistics catalogue	Yes	npx tsx packages/cli/src/cli.ts probe bts-transportation-stats
estimatetax-income-tax	EstimateTax US federal, state, and local income tax	Yes	npx tsx packages/cli/src/cli.ts probe estimatetax-income-tax
nyc-open-data-catalogue	New York City Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe nyc-open-data-catalogue
callook-callsigns	US amateur radio callsign records through Callook.info	Yes	npx tsx packages/cli/src/cli.ts probe callook-callsigns
college-scorecard-institutions	College Scorecard institution outcomes, US Department of Education	Yes	npx tsx packages/cli/src/cli.ts probe college-scorecard-institutions
publicdatahub-hospital-quality	CMS hospital quality measures, republished by PublicDataHub	Yes	npx tsx packages/cli/src/cli.ts probe publicdatahub-hospital-quality
nextbus-transit-vehicles	Portland Streetcar live vehicle positions through the NextBus XML feed	Yes	npx tsx packages/cli/src/cli.ts probe nextbus-transit-vehicles
federal-register-documents	Office of the Federal Register, published documents	Yes	npx tsx packages/cli/src/cli.ts probe federal-register-documents
nhtsa-vehicle-models	NHTSA vehicle models by model year and make	Yes	npx tsx packages/cli/src/cli.ts probe nhtsa-vehicle-models
fcc-socrata-catalogue	FCC Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe fcc-socrata-catalogue
ny-state-open-data-catalogue	New York State Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe ny-state-open-data-catalogue
chicago-open-data-catalogue	Chicago Data Portal catalogue	Yes	npx tsx packages/cli/src/cli.ts probe chicago-open-data-catalogue
seattle-open-data-catalogue	Seattle Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe seattle-open-data-catalogue
texas-open-data-catalogue	Texas Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe texas-open-data-catalogue
colorado-open-data-catalogue	Colorado Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe colorado-open-data-catalogue
washington-state-open-data-catalogue	Washington State Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe washington-state-open-data-catalogue
michigan-open-data-catalogue	Michigan Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe michigan-open-data-catalogue
oregon-open-data-catalogue	Oregon Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe oregon-open-data-catalogue
pennsylvania-open-data-catalogue	Pennsylvania Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe pennsylvania-open-data-catalogue
illinois-open-data-catalogue	Illinois Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe illinois-open-data-catalogue
new-jersey-open-data-catalogue	New Jersey Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe new-jersey-open-data-catalogue
vermont-open-data-catalogue	Vermont Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe vermont-open-data-catalogue
los-angeles-open-data-catalogue	Los Angeles Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe los-angeles-open-data-catalogue
san-francisco-open-data-catalogue	San Francisco Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe san-francisco-open-data-catalogue
california-open-data-catalogue	California Open Data catalogue	Yes	npx tsx packages/cli/src/cli.ts probe california-open-data-catalogue

Each adapter knows how to fetch live data and parse the response. Tests load a committed fixture; a live call throws instead of falling back to one.

Adapter examples

A probe prints a JSON summary with the probe id, name, auth, an ok or status line, and a sample of the live data.

# BLS - the national unemployment rate, monthly, seasonally adjusted
npx tsx packages/cli/src/cli.ts probe bls-unemployment-rate

# USGS - earthquakes of magnitude 2.5 and above near the Hawaiian islands
npx tsx packages/cli/src/cli.ts probe usgs-hawaii-earthquakes

Language-agnostic access

HTTP API

npm install
npm run dev:api        # http://localhost:8787

• GET /health - health check
• GET /metrics - request counts in Prometheus format
• GET /openapi.json - machine-readable OpenAPI spec (generate clients in any language from this)
• GET /docs - Swagger UI
• GET /api/sources - list every adapter
• GET /api/sources/:id/probe - live probe one source
• GET /api/sources/:id/data - fetch and parse the live payload for one source

curl 'http://localhost:8787/api/sources/bls-unemployment-rate/data'

CLI

npx tsx packages/cli/src/cli.ts sources
npx tsx packages/cli/src/cli.ts probe bls-unemployment-rate

Output goes to stdout as JSON, errors go to stderr, and the exit code is 0 on success.

Quick start (TypeScript)

npm install
npm run check

import { probeAllUsDataSources } from '@usa-open-data-connectors/usa-sources';

const probes = await probeAllUsDataSources();
console.log(probes.map((probe) => `${probe.id}: ${probe.ok ? 'ok' : probe.status}`).join('\n'));

Environment variables

Variable	Needed for	Where to get it
SENTRY_DSN	Error tracking (optional, off by default)	sentry.io

No registered connector needs a key. A future keyed adapter reads its key from an environment variable read at process start, server-side.

Testing

npm run check          # format, lint, type-check, and unit tests with coverage
npm run test:smoke     # live smoke tests against the real APIs (needs RUN_SMOKE=1)

Unit tests use committed fixture snapshots pulled from the live APIs, so they run offline. Smoke tests are opt-in via RUN_SMOKE=1 and hit the real endpoints. They need no key.

Other languages

The earlier Python and Ruby ports cover the adapter ids they shipped with. The new NIH RePORTER adapter is TypeScript-only for now. Each ported adapter has a live fetcher, a parser, a committed fixture, and an offline test. Both ports use the shared fixtures in packages/usa-sources/src/fixtures, so a snapshot is stored once.

Python lives in python/src/usadata and Ruby lives in ruby/lib/usadata. Both are keyless and expose US_DATA_SOURCES, get_us_data_source, and probe_all_us_data_sources. Use them from a source checkout:

import sys
sys.path.insert(0, "python/src")

from usadata import US_DATA_SOURCES
print([source.id for source in US_DATA_SOURCES])

$LOAD_PATH.unshift("ruby/lib")
require "usadata"

puts Usadata::US_DATA_SOURCES.map(&:id)

Run the offline port tests with:

PYTHONPATH=python/src python -m pytest python/tests
ruby -I ruby/lib -I ruby/spec ruby/spec/sources_spec.rb

The HTTP API and usdata CLI remain the simplest option from R, Julia, or any other language. Both print or serve JSON.

Run the API in Docker

The Dockerfile at the repo root runs the HTTP API on port 8787 with a non-root user and a health check.

docker build -t usa-connectors .
docker run -p 8787:8787 --env-file .env usa-connectors

Any keys come from the environment only. Every registered endpoint works keyless.

Documentation

• docs/ARCHITECTURE.md - how the pieces fit together, in plain language
• docs/CONNECTOR_DISCOVERY.md - every adapter, the live check behind it, and the sources not yet probed
• docs/SECURITY.md - key handling and the security checklist
• docs/GLOSSARY.md - plain-language definitions of every term
• docs/RELEASING.md - how versions, tags, and publishing work
• docs/AGENT_CONTEXT.md - handoff context for an agent working in this repo
• docs/faq.md - keys, licences, fixtures, and how to add a source
• docs/contact.md - how to report a bug, a dead source, or a security problem
• COUNTRY.md - adapter status table and the sources still to come

The other files under docs/ come from the shared project template (olitreadwell/template) and describe the template's own gates. For this repo, the API contract lives at GET /openapi.json and /docs.

Contributing

See CONTRIBUTING.md for how to set up the repo, run the checks, and open a pull request.

License

MIT