My AI assistant and I share the same brain. Not metaphorically, we literally read and write to the same knowledge base. When I update a project note in Emacs, Nabu (my AI) sees the change. When Nabu logs something it learned, I see it in my daily file. This post is about how it works.

How It Works#

Nabu connects to my self-hosted Matrix server and it has access to my org-roam knowledge base via an MCP server I built. But it’s not just access, it’s integration. Nabu treats org-roam as its source of truth:

I had an NVIDIA Jetson Orin Nano sitting around collecting dust, along with a 5-inch touchscreen monitor, and thought it would be fun to build a voice interface for my second brain. Not because I needed one, but because the idea of talking to a device with an oscilloscope-style voice visualization seemed like a cool weekend project.

Why Build a Cyberdeck?#

Honestly, I just thought it would be neat. I had the parts, I had the time, and the concept of a voice-first chat device in a custom enclosure sounded fun to build. I had no productivity goals, no problem to solve, just a project for the sake of building something interesting.

I’ve been building what I call my “Second Brain,” an Emacs-based knowledge management system that doesn’t just store notes, but actively helps me think. It’s built on org-roam, extended with semantic search via vector embeddings and proactive surfacing of relevant information. The goal isn’t to replace my thinking but to augment it, surfacing connections I might miss and nudging me toward follow-ups I’ve forgotten.

Why Emacs and Org-roam#

Before diving into what I’ve built, it’s worth explaining the foundation I chose.

Two AMD MI60s from eBay cost me about $1,000 total and gave me 64GB of VRAM. That’s enough to run Llama 3.3 70B at home with a 32K context window.

When I started looking into running large language models locally, the obvious limiting factor was VRAM. Consumer GPUs top out at 24GB, and that’s on an RTX 4090 at the high end. I wanted to run 70B parameter models locally, on hardware I own.