AI Companion

A single dedicated AI friend for your Minecraft world — not a static NPC, but an active participant. It has objectives, tells friend from threat, autonomously navigates, gathers and crafts, builds structures on request, and fights alongside you. Its "voice and judgment" come from a large language model that you run — a local llama.cpp server on your own machine, or any hosted OpenAI-compatible API (e.g. xAI/Grok, OpenAI) if you'd rather trade privacy for frontier-model quality.

The LLM decides what to do and what to say. The heavy lifting — pathfinding, task execution, world interaction — lives in engine code and keeps working even if the LLM is offline.

⚠️ Scope — singleplayer and LAN, not public servers

This is an alpha, and it is built and tested for singleplayer and Open to LAN worlds played with people you trust. It is not ready for a public multiplayer server, and I'd rather tell you why than let you find out:

• The companion is not a player, so land-claim and protection mods can't see it. Claim mods hook player-specific block-break events; a companion is a LivingEntity and never fires them. On a protected server it could plausibly dig through claimed land.
• One companion identity, server-wide. Name and persona come from a single server config file, so every player would get the same character.
• Replies go to everyone. The companion's chat lines are sent to all online players regardless of distance.
• Cost caps are per-server, not per-player. One person's conversation spends everyone's budget.

Fixing these properly is the 1.0 milestone. Until then: singleplayer and trusted LAN. On a dedicated server the /companion command is op-gated, so it won't surprise you.

⚠️ Requirements — read this first

This mod does nothing on its own. It needs an OpenAI-compatible chat-completions endpoint to think. You provide one of:

• Local (private, free): a running llama.cpp server (or any local OpenAI-compatible server) on your machine/LAN. Nothing leaves your network.
• Cloud (frontier quality, paid): a hosted OpenAI-compatible API and an API key (e.g. xAI/Grok). A per-session request cap is built in so a runaway loop can't rack up spend.

Plus: Minecraft 1.20.1, Fabric Loader, and Fabric API. The pathfinding/task engine is bundled inside this jar — you do not need a separate download for it.

Voice output (the companion speaking aloud) is optional and needs a separate local TTS service (a small Kokoro Docker container). It is not part of this download — grab the tts/ folder from the GitHub repo and run docker compose up -d. Full steps in the repo's tts/README.md. See Configuration to enable it.

Features

• Autonomous agent, not a script — mines, collects, crafts, engages in combat, and builds structures via a proven task engine (Automatone/Baritone pathfinding + an AltoClef-derived task layer).
• Your brain, your rules — one config file points the companion at a local llama.cpp server or a hosted frontier API. Swap between them anytime.
• Survives the LLM going away — navigation and task execution are engine-side; if the model is offline or slow, the companion doesn't freeze mid-world.
• Persona & identity — set the companion's name, description, and personality in config; the persona is injected into a hardened prompt scaffold (it shapes voice, it doesn't get to override safety structure).
• Held tools & weapons — the companion visibly wields tools/weapons in its main hand, and its held item is part of what the LLM "sees," so equips are confirmable.
• Reliable command output — tolerant JSON parsing with graceful fallback: a malformed model reply is spoken as chat instead of dropping the turn.
• Know what you're spending — a live HUD panel shows the session's token total, in/out split, and a tokens-per-minute graph for the last 30 minutes, so a paid endpoint never surprises you and a runaway companion is obvious within a minute (/companion tokens hides it if you'd rather not see it). The running total is also reported to chat and the log every 100k tokens. Optional hard caps and a chat trigger prefix are there if you want them; both are off by default.
• Dies like a player — if the companion is killed it drops everything it was carrying, armour included, and tells you where. Nothing it gathered or you handed it is lost to the death; go and pick it up like you would your own. (Items still despawn on the usual five-minute timer, so don't dawdle.)
• Find it again — a locator bar HUD points toward your companion and works past entity-tracking range, so wandering off isn't a lost companion.
• Recall commands — call a wandered-off companion back, or ask where it is.
• In-game config screen — /companion config opens a settings UI (identity, LLM, voice, behavior) right in the client; edits apply live, no restart. /companion reload re-reads the JSON file if you prefer editing by hand. With Mod Menu installed you also get a config button there (optional — nothing breaks without it).
• Optional local voice — route spoken lines to a local Kokoro TTS endpoint; only the companion's spoken message is ever voiced, never its reasoning or commands.

Commands

Beyond commands, just talk to it in chat — that's the primary way you direct it.

Configuration

On first launch the mod writes config/aicompanion.json with documented defaults. You can edit it two ways: the in-game screen (/companion config, or the Mod Menu gear button) which applies changes live, or the JSON file by hand followed by /companion reload. Key settings:

• Identity: companion name, description, persona.
• LLM: endpoint (default http://localhost:3030), model, temperature, maxTokens, request timeout.
• Cloud/frontier: set endpoint to a hosted API and supply the key via the AICOMPANION_LLM_APIKEY environment variable (preferred) or the apiKey field. Worked example for xAI/Grok:

"llm": {
  "endpoint": "https://api.x.ai",
  "model": "grok-4-1-fast-non-reasoning",
  "temperature": 0.7,
  "maxTokens": 200,
  "apiKey": "xai-your-key-here"
}

endpoint is the base URL only — no trailing slash and no /v1; the mod appends /v1/chat/completions itself. And pick a non-reasoning model: reasoning models are slower and bill you for thinking tokens the companion never uses. Any other OpenAI-compatible provider works the same way.

Using OpenAI? Prefer gpt-4.1-nano, gpt-4o-mini, or gpt-4.1. The gpt-5.x and o-series models are reasoning models whose hidden thinking counts against maxTokens — at the default 200 they can burn the entire budget thinking and return an empty reply with no error, so the companion simply goes quiet. Give them maxTokens of 1000+ or stay on a non-reasoning model.

• Spend awareness: llm.usageReportEveryTokens (default 100000) prints a running token total to chat; 0 silences it. llm.maxRequests is a separate, opt-in hard cap that makes the companion stop responding once hit — leave it at 0 unless you want a hard stop.
• Chat gating: behavior.triggerPrefix (blank by default) makes the companion answer only messages starting with that prefix, so ambient chat costs nothing. behavior.thinkThrottleSeconds sets a minimum gap between LLM turns — messages inside the window are queued, not dropped.
• Survival-honest building: the companion pays for what it builds out of its own inventory, one item per block, instead of conjuring blocks. Short of something? It goes and collects it, then builds — you just ask once. Set behavior.buildCostsMaterials to false for creative-style building.
• Builds land where you asked: every plan is checked against the real terrain before a block is placed, so a structure can't end up buried out of sight — but "put it on top of the ground" still means on top, and towers still go up. Only a plan hallucinated far into the sky is refused, and it costs no materials. Rebuilding something that's already there spends nothing and says so instead of pretending. behavior.buildGroundCheck turns the check off.
• Voice (optional): enable TTS and point it at a local Kokoro endpoint (client-side playback).

What's bundled

This download is self-contained. It nests the mod's forked PlayerEngine (pathfinding + task engine) and the Cloth Config library (for the settings screen) inside its own jar — no separate downloads.

Do not also install a standalone PlayerEngine jar in your mods/ folder — two copies of the engine will collide on load.

Credits & license

This mod is a fork/consumer of the excellent work of others, and is distributed under the GNU LGPL-3.0 in keeping with its upstream. Full credit:

• PlayerEngine by Goodbird-git — LGPL-3.0 — the framework this bundles and builds on.
• Automatone, a fork of Baritone (leijurv & contributors) — LGPL-3.0 — pathfinding/navigation.
• The task engine descends from the AltoClef (adris.altoclef) lineage — mine/collect/craft/combat/build.
• Brain integration adapted the LLM seam from PlayerEngine's Player2 path to a local/OpenAI-compatible endpoint. The reference consumer Player2NPC was studied for structure only; no Player2NPC source is reused (it carries no license).

Source code: the complete source for this mod and its modified engine is available at https://github.com/adevivo/ai-companion — required by, and provided under, the LGPL-3.0.

Not affiliated with, or endorsed by, Mojang or Microsoft. "Minecraft" is a trademark of Mojang Synergies AB.