Tulpa

LLM-powered AI companions for Minecraft

Summon a companion, then just talk to it. A real player's body — driven by a mind that runs on your machine, on your own API key.

Tell your companion "go mine me a stack of iron," "build a hut where I'm standing," "follow me and kill anything hostile," or "smelt these and put the bars in that chest" — a large language model plans the work and drives the companion through it: pathfinding across terrain, mining, building, fighting, crafting, and using containers, entirely on its own. Talk to it in any language your model speaks.

Features

• 🗣️ Just talk to it. Natural-language requests become real, multi-step work in the world — no scripts, no GUIs full of toggles.
• 🧠 The brain runs on your machine. The agent loop calls the LLM from the owner's client with the owner's API key. No server-side LLM, no shared bill — each player pays only for their own companion's thinking.
• 🧍 The body is a real player, not a custom mob. Companions are server-side fake players, so they interact with the world through genuine player code — they "just work" with other mods, redstone, mob AI, and containers instead of needing per-feature adapters.
• 📖 Teach it any mod with a Markdown file — no adapter, no waiting. Because the companion is a real player driving universal tools (transfer and interact_at work on any GUI or block, vanilla or modded), making it play a new mod doesn't take a compatibility patch — it takes a SKILL.md written in plain English. Want it to run a Create andesite farm or progress through Mekanism? Describe the workflow and drop the file in. You don't wait on the mod author, and you don't wait on us. → Skills
• 🧭 Baritone-grade pathfinding, from scratch. A self-contained A* pathfinder reimplements Baritone's cost model and movement set 1:1 (walk, diagonal, ascend/descend, jump, parkour, pillar-up, dig-down, scaffold-bridging) — no Baritone dependency.
• 🛠️ 27 tools. Mine, build, fight (native melee + bow), craft by hand-placing into the grid, use any container, scan the world, locate structures/biomes, manage inventory, and plan with a live to-do list.
• 🔌 Bring your own model. OpenAI, DeepSeek, Kimi/Moonshot, MiniMax, Doubao, and Qwen out of the box — any OpenAI-compatible endpoint via a custom base URL.
• 🪶 No third-party SDKs. LLM transport is the JDK's built-in HttpClient + Gson. That's it.
• 🧩 One codebase, both loaders. Ships for Fabric and NeoForge (MC 26.1.2, Java 25).

Quick start

1. Install the mod (plus Fabric API if you're on Fabric) and launch the game once to generate the config.
2. Add your API key. Press G → Settings (or run /tulpa settings), pick your provider, and paste your key.
3. Summon a companion: /tulpa player summon <name>
4. Press G to open the chat and tell it what to do.

You:   挖一组铁矿回来 / go grab me a stack of iron
Tulpa: On it. Heading underground to find iron.
       ▸ 4 steps · locate_biome · move_to · auto_mine · collect_items   ✔
Tulpa: Got 64 raw iron — want me to smelt it?

Usage

Open a companion with G (it goes straight to chat if you only have one; otherwise it shows the roster). The panel has three tabs:

A left-edge HUD shows a small avatar per companion: idle ones tuck away to a thin gold sliver; when one speaks, its avatar and a speech bubble slide out together.

Commands

How it works

   You (owner's client)                          Server
   ─────────────────────                         ──────────────────────────
   chat box ─▶ agent loop ──▶ LLM API
                   │  (your key, your model)
                   │  tool calls
                   ▼
             ExecuteToolPayload ───────────▶  task queue on the fake player
                   ▲                               │  per-tick dispatcher
                   │   results                     ▼
             TaskResultPayload ◀──────────   pathfinding · mining · combat ·
                                             crafting · container use …

1. You type a message; the client-side agent loop sends the conversation + tool schemas to your chosen LLM and streams the reply.
2. Read-only tools (status, scans, recipe lookups) answer instantly; world-action tools are shipped to the server, queued on the companion's body, and run by a per-tick task dispatcher.
3. Results flow back into the conversation, so the model sees what happened and decides the next step. It can chain many actions per request — and you can hit Stop anytime.
4. History is auto-compacted (summarized) as it approaches the context window, so long sessions keep working.

The companion

• Persists like a player — state lives in a vanilla playerdata/*.dat (position, inventory, health, owner); companions return on login.
• Owned by you, across dimensions — ownership is stored and checked by UUID (vanilla getOwner() is level-scoped and breaks across dimensions, so Tulpa avoids it).
• Recoverable death — vanilla death runs normally (drops, keepInventory, grave mods all work); the agent pauses and the body respawns at the owner after a short delay.
• Shows up as a player in the tab list and renders with its own skin.

What it can do

The model expresses intent — the companion figures out how (pathing, tool choice, reach) on its own.

All 27 tools

Movement & navigation

• move_to — pathfind to coordinates (bridges, digs, jumps as needed)
• locate_structure — find the nearest structure (village, stronghold, …)
• locate_biome — find the nearest biome

Mining & building

• auto_mine — "gather N of these blocks" — finds, paths to, mines, repeats
• break_block — remove one exact block
• place_block — place a block at an exact spot (optional facing)

Combat (native player combat — real cooldowns, weapon mods, crits)

• hunt — melee N mobs of given types
• shoot — kill N mobs with a bow (charges, leads moving targets)

Inventory & items

• collect_items — pick up nearby drops
• drop_items — drop items (hand off to you / shed junk)
• equip_item — wear/wield gear
• eat_item — eat food to heal

Containers & interaction

• interact_at — aim at and use a block (open chests, furnaces, doors, levers…)
• interact_entity — follow and use a moving entity (villagers, animals…)
• transfer — move items between slots of an open GUI by intent
• inspect_gui — read the open container's slots & machine progress
• close_gui — close the container

Crafting & recipes

• lookup_recipe — JEI-style recipe lookup (crafting / smelting / stonecutter / smithing)
• (crafting itself is done by hand-placing items into the grid via transfer + interact_at)

Perception & status

• get_self_status · get_owner_status · get_world_info
• scan_blocks · scan_nearby_entities · inspect_block

Planning & meta

• todowrite — the model's own to-do list (shown live in the plan panel)
• load_skill — pull in a Markdown skill (a saved workflow) full-text, on demand
• wait — idle deliberately (smelting, nightfall, timers) instead of burning tokens

Skills — teach it anything, in Markdown

Tulpa's interface to the world is natural language, the whole way down. So is the way you extend it. A skill is just a SKILL.md file — YAML frontmatter (name + description) and a plain-English how-to in the body. No code, no Java, no recompile.

This is the payoff of the fake-player design. The companion already touches the world through universal tools — transfer and interact_at drive any container, machine, or block exactly like a human hand, modded or not. So teaching it a new mod is never an integration project; it's a writing task. You don't wait for the mod author to add support, and you don't wait for us to ship a patch — you write down the workflow and the companion can do it.

config/tulpa/skills/create_andesite_farm/SKILL.md
─────────────────────────────────────────────────
---
name: create_andesite_farm
description: Build and run a Create andesite generator — assemble the
  kinetic chain, point it at a depot, and stockpile the andesite.
---
# Skill: create_andesite_farm

1. Craft a water wheel and an andesite casing (lookup_recipe each).
2. Place the wheel over flowing water so it spins; ...
3. ... (the steps, in your own words)

Drop that folder in and the companion can play Create — the same trick teaches it Mekanism, Farmer's Delight, Applied Energistics, or your own modpack's progression, the moment you can describe it.

How the model uses them — progressive disclosure, so the prompt stays lean:

• Every turn, the model sees only a table of contents: each skill's name + description, nothing more. Dozens of skills cost almost no tokens.
• When a task matches a description, the model calls the load_skill tool and the full body streams into the conversation — just-in-time instructions, paid for only when they're actually relevant.

Where they live: config/tulpa/skills/<name>/SKILL.md. Tulpa ships a built-in set — combat basics, the container/crafting/smelting playbook, and a full Nether → blaze rods → ender pearls → stronghold → dragon speedrun chain — extracted to that folder on first launch. After that the folder is yours: edit them, delete them, write your own. It's the same SKILL.md shape the broader agent-skills ecosystem uses, so skills written elsewhere mostly drop straight in.

Configuration

Set everything in-game (G → Settings), or edit the config file directly.

Config lives in config/tulpa.json (Fabric) or config/tulpa-common.toml (NeoForge).

Building from source

A standard Gradle multiloader project: shared code in common/, thin loader modules in fabric/ and neoforge/. The common module reaches loader-specific functionality through a small Services abstraction (config, networking, platform) resolved via ServiceLoader.

./gradlew build                 # build both loaders
./gradlew :fabric:runClient     # run the Fabric client
./gradlew :neoforge:runClient   # run the NeoForge client

Stack: Java 25 · Minecraft 26.1.2 · Fabric Loader 0.19.2 / Fabric API 0.148.2 · NeoForge 26.1.2.50-beta · Mixin + MixinExtras · LLM transport via java.net.http.HttpClient + Gson (no third-party HTTP/LLM libraries). Only three small mixins are used — the design leans on the fake-player approach instead of invasive patches.

License

Released under the MIT License — do whatever you like (use, modify, distribute, even commercially); just keep the copyright notice.

Built on the MultiLoader Template.