Llmgine

The LLM-native game engine. An ECS game engine where intelligence is a core primitive: attach a Mind (LLM cognition), Eyes (perception/vision), and Voice (neural TTS) to any entity the same way you attach physics or a sprite. An NPC, a boss, a monster, a quest giver, a faction, the weather — if it's in the world, it can think, see, and speak.

Working title. TypeScript · 3D (three.js) + headless + 2D canvas · MIT.

deterministic 60 Hz ECS sim  ←— validated intents —— async LLM minds
        │                                                  ▲
        └—————— perception snapshots + pixel vision ———————┘

The hard problem: games are deterministic real-time loops; LLMs are slow, async, and non-deterministic. llmgine resolves it structurally:

• The sim never blocks on a thought. Minds observe snapshots, think on their own cadence (plus event wakeups: damaged, spoken to), and return intents.
• Intents pass the same validated action pipeline as player input — a Mind can only do what its body allows. No hallucinated teleports, no 9999 damage.
• Every LLM-augmented module has a deterministic fallback (behavior policies, weighted loot tables, quest state machines). If the API is down, the game still runs.
• Genesis turns the LLM into a content generator: prefabs, loot, quests as validated JSON. The model proposes; the engine disposes.

Quickstart

Requires Node >= 18. llmgine is not published to npm yet — today you run it from a clone (which works end-to-end); npm install llmgine will work once it's published.

From a clone (the working path today):

git clone https://github.com/lordbasilaiassistant-sudo/llmgine
cd llmgine
npm install
npm run build       # engine → dist/
npm test            # unit tests, no network (pretest rebuilds dist/ automatically)
npm run demo        # the 3D arena demo → http://localhost:4173

Optional: copy .env.example to .env and add a ZAI_API_KEY to give minds a live model (see below) — the demo auto-detects it locally.

Start your own game — scaffold from the clone; the generated project links back to it with a file: dependency (no npm registry needed):

node dist/cli/index.js create my-game    # from the clone root
cd my-game && npm install && npm run dev # http://localhost:4173

From npm (once published):

npm install llmgine        # not yet — 404s today, tracked for a deliberate release
npx llmgine create my-game

What the code looks like:

import {
  World, GameLoop, SpatialGrid, ActionRegistry, actionSystem,
  Transform, Velocity, Named, Health, Speech, Behavior,
  STANDARD_VERBS, behaviorSystem, movementSystem,
  Mind, MindMemory, CognitionDriver, OpenAICompatibleProvider,
} from "llmgine";

const world = new World(42);            // seeded — deterministic
const grid = new SpatialGrid();
const actions = new ActionRegistry();
for (const v of STANDARD_VERBS) actions.register(v);

// any entity + Mind = intelligent entity
const guard = world.create();
world.add(guard, Transform, { x: 0, y: 0 });
world.add(guard, Velocity);
world.add(guard, Named, { name: "Gate Guard" });
world.add(guard, Health);
world.add(guard, Speech);
world.add(guard, Behavior, { mode: "idle" });
world.add(guard, Mind, {
  persona: "A vigilant town guard. Suspicious of strangers.",
  goals: ["guard the gate"],
  thinkEvery: 8,                        // seconds between thoughts
  fallbackMode: "wander",               // deterministic policy if the LLM is unavailable
});
world.add(guard, MindMemory);

const driver = new CognitionDriver({
  provider: new OpenAICompatibleProvider(),   // reads ZAI_API_KEY / LLM_API_KEY + LLM_BASE_URL
  actions, grid,
});
world.addSystem(actionSystem(actions));
world.addSystem(behaviorSystem());
world.addSystem(movementSystem(grid));
world.addSystem(driver.system());

new GameLoop(world).start();            // browser; or loop.advance(n) headless

Get a free model (GLM)

The default provider targets z.ai's OpenAI-compatible API, where glm-4.5-flash is free — free minds for every NPC in your game:

1. Create a key at z.ai and set ZAI_API_KEY.
2. That's it. Tiers: fast (flash — NPC chatter), smart (deep reasoning), vision (pixel Eyes). Map tiers to any models you like.

Any OpenAI-compatible endpoint works instead: OpenAI, Ollama, LM Studio, vLLM — new OpenAICompatibleProvider({ baseUrl, apiKey, models }).

Disclosure: if you want more than the free tier, this is a referral link for the GLM Coding Plan — we may receive credit, which funds the project's development: https://z.ai/subscribe?ic=BWTG6TRYYQ

The demo — The Neural Colosseum

A 3D torchlit arena: you (a gladiator) vs The Arena Master, a boss whose mind is a live GLM model. It perceives the pit, taunts you in character (rendered in the "thought ribbon" and speech bubbles, voiced by local Kokoro neural TTS), commands its goblins, fights, and drops loot through deterministic tables. Remove the API key and the same fight runs on pure instinct.

git clone https://github.com/lordbasilaiassistant-sudo/llmgine
cd llmgine && npm install
npm run demo        # http://localhost:4173 — auto-detects ZAI_API_KEY locally

For AI agents: the MCP server

The engine ships as an MCP tool so agents can build and test games headlessly. Claude Code auto-connects via the repo's .mcp.json when opened inside a built clone; any other client:

{ "mcpServers": { "llmgine": { "command": "node", "args": ["<path-to-clone>/dist/mcp/server.js"], "env": { "ZAI_API_KEY": "…" } } } }

Tools: create_world, define_prefab, define_loot_table, list_prefabs, spawn, attach_mind, act, run (advance N ticks → event log), query_world, save_world, load_world, destroy_world, generate_prefab (Genesis). An agent can design a boss, attach a mind to it, simulate 10 seconds of combat, and read the death/loot events back — no browser, no human in the loop. Full walkthrough: docs/mcp.md.

Build (and play) games with your agent

Agents are first-class players here, not just builders. Every game can wire an AgentPort (llmgine/agent) — observe/act/step/save through the exact same Eyes-perception + validated-verb pipeline the LLM Minds use. In a browser it's window.llmgine; with the dev server running, any local process can drive the live game over HTTP:

curl -s localhost:4173/agent/call -d '{"method":"observe"}'
curl -s localhost:4173/agent/call -d '{"method":"act","args":["move_to",{"x":0,"y":-100}]}'
curl -s localhost:4173/agent/call -d '{"method":"step","args":[120]}'   # deterministic 2s
curl -s localhost:4173/agent/call -d '{"method":"actionLog"}'           # why was my verb rejected?

step() pauses real time and advances the fixed-timestep sim — agent playtests are reproducible. Rejected actions carry the validator's reason.

Give your agent the skill file at skills/llmgine/SKILL.md (drop it into .claude/skills/ for Claude Code) — it teaches the architecture contract, the build loop, the gotcha ledger, and the verify loop. npm run agent:verify runs the headless engine acceptance (determinism, adversarial verb rejection, LLM-down fallback) in seconds.

What's in the box

Layer	Contents
core	ECS, fixed-timestep loop, seeded RNG, event journal, spatial grid, prefabs (validated JSON), action/intent pipeline, save/load
gameplay	combat (PvE/PvP, factions, aggro), loot/drop tables, quests + rewards, inventory, dialogue/speech, spawning — each fully functional with zero LLM
ai	provider-agnostic inference (tiers: fast/smart/vision), Mind/Eyes/Voice components, cognition scheduler, memory, budgets + caching, Genesis content generation
render3d	three.js renderer: model factories with live-sim-driven animation, chase cam, capture() for pixel vision
render	headless renderer (tests/servers/MCP) + canvas 2D (prototyping/minigames)
mcp	the engine as an agent tool

Full design: ARCHITECTURE.md. Focused guides in docs/: input (touch + gamepad) · audio · save/load · navigation · projectiles · glTF models · MCP server.

Honest status (v0.1)

Works (tested): everything in the table above — 50 unit tests + a live GLM suite (npm run test:live) where a real model drives a Mind through the intent pipeline and Genesis generates a valid, spawnable prefab. The demo has been played to completion by a scripted agent in a real browser: pack culled, boss duel won, quest completed, rewards granted, live GLM taunts mid-fight — and the same run with no API key completes on deterministic fallbacks.

Also in the box now (all tested): touch joystick + gamepad input, procedural SFX + looping ambient music (zero asset files), verb-gated projectiles/ranged combat, NavGrid A* pathfinding (behavior routes around obstacles), save slots (F5/F9 quicksave in the demo), glTF model helpers, and a provider-level repair for GLM flash's malformed tool calls (captured live, unit-tested).

Also works (tested): llmgine create <name> (run from a clone: node dist/cli/index.js create <name>) scaffolds a starter game that installs and builds against the clone via a file: link — proven end-to-end (create → npm install → build). llmgine export windows|android|ios|pwa|store generates the Electron/.exe config, Capacitor mobile config, installable PWA, and a store listing kit (assets checklist, pricing worksheet, AI disclosure) — generator output is covered by CLI subprocess tests. The heavy toolchains run in your game project (#1 tracks producing reference artifacts).

Remaining gaps (tracked as issues):

• Human feel-tests: audio ear-test (#2), phone/pad in hand (#3), a real .glb exercised in an example (#6).
• Multiplayer (the event journal + intent log are the designed foundation).
• Voxel/heightmap terrain; STT input.
• Vision ("pixels" Eyes) is wired end-to-end but not yet exercised by the demo.

License

MIT. Contributions welcome — see CONTRIBUTING.md.