Tool Hut

Founder stack playbook

Run a fully local and private AI stack

Local chat, document Q&A, embeddings, and transcription with no routine content sent to a hosted AI provider.

Who it's for

A privacy-sensitive solo builder with suitable hardware who accepts lower model quality and more maintenance in exchange for local control.

Constraints

Single-machine or home-lab deployment, no automatic cloud fallback, encrypted backups, and model licences reviewed before commercial use.

Approximate monthly cost

US$0 in software subscriptions; electricity, storage, backups, and suitable hardware are the real costs. A capable new machine can cost US$1,000-3,000+ upfront.

How the stack works

  1. Run quantised models through Ollama and bind its API to localhost or a private network only.
  2. Use Open WebUI for general chat and AnythingLLM for workspace-scoped document retrieval.
  3. Store embeddings in local Qdrant and keep original documents on encrypted local storage.
  4. Transcribe audio with whisper.cpp before it enters the retrieval pipeline.
  5. Pin model and container versions, back up configuration, and test restore without enabling telemetry-heavy cloud integrations.

The stack

Local model runtime

Ollama

A small CLI and local API make downloading, pinning, and serving common open models straightforward.

Skip if: you need maximum inference tuning, unsupported accelerators, or production multi-node serving.

Alternatives: llama.cpp, LocalAI, LM Studio

Chat interface

Open WebUI

It provides a polished self-hosted chat surface over Ollama without building account and conversation UI.

Skip if: the machine is single-user and a desktop client is safer and simpler than a web service.

Alternatives: LM Studio, Jan, Msty

Local document workspace

AnythingLLM

Workspace-level document ingestion and local-provider support deliver useful private RAG with little assembly.

Skip if: retrieval behaviour must be deeply customised, evaluated, or embedded inside your own product.

Alternatives: PrivateGPT, Open WebUI knowledge, LlamaIndex

Local vector store

Qdrant

A local container gives production-grade filtering and vector search without sending embeddings off-device.

Skip if: the document tool's built-in store is sufficient or another database would only duplicate local state.

Alternatives: Chroma, pgvector, Weaviate

Container runtime

Docker

Pinned containers make the UI, vector store, and supporting services reproducible and easier to restore.

Skip if: Docker Desktop licensing, overhead, or GPU integration is worse than native services on the target machine.

Alternatives: Podman, Native system services, Kubernetes for a larger lab

Local transcription

Whisper.cpp

Efficient on-device transcription keeps meeting and voice data inside the same privacy boundary.

Skip if: real-time latency, diarisation, or noisy-audio accuracy requires a specialist hosted speech API.

Alternatives: Whisper, Deepgram, AssemblyAI

Low-level inference engine

llama.cpp

It offers fine control and broad hardware support beneath many local runtimes when defaults stop being enough.

Skip if: Ollama already supports the required model and hardware; operating both can create duplicate model stores and confusion.

Alternatives: Ollama only, LocalAI, KoboldCpp

Verification note: Tool availability and licences checked 12 July 2026. Local does not automatically mean secure: disk encryption, network binding, access control, backups, model provenance, and telemetry settings remain the operator's responsibility.

← Browse all founder playbooks