Advertisement

Running an AI assistant on your own hardware has gone from a research project to something you can set up in minutes. Open-weight models from several labs, including Meta’s Llama, Google’s Gemma, Alibaba’s Qwen, and Mistral, can be downloaded and run locally, and a set of mature open source tools handles the hard parts: fetching models, using your GPU, and giving you a chat interface or an API. The result is an assistant whose prompts and documents never leave your machine, with no subscription and no usage limits.

The tools below fit together more than they compete. llama.cpp is the inference engine underneath much of the ecosystem, and Ollama wraps local models in a simple command line and API. Jan and AnythingLLM are complete apps for chatting and for working with your documents, and LibreChat turns a server into a shared, multi-user chat service. How capable a local assistant feels depends mostly on your hardware and the size of the model you can run.

Advertisement

Our 5 Picks

  1. Jan

    Most people: a desktop chat app for local models

    Jan is a free, Apache-2.0 desktop app for chatting with AI models that run on your own computer, with optional connections to cloud providers.

    Jan: features, platforms, and license →

  2. Ollama

    Running local models for the command line and other apps

    Ollama is a free, MIT-licensed tool for downloading and running open-weight LLMs locally, with a command line and an API other apps connect to.

    Ollama: features, platforms, and license →

  3. LibreChat

    A self-hosted, multi-user chat service for teams

    LibreChat is a free, MIT-licensed, self-hosted AI chat app with multi-user accounts, agents, and support for Claude, OpenAI, Google, and local models.

    LibreChat: features, platforms, and license →

  4. llama.cpp

    Maximum control and performance for technical users

    llama.cpp is an MIT-licensed C/C++ engine for running LLMs locally on CPUs and GPUs, with a command-line chat tool and an OpenAI-compatible server.

    llama.cpp: features, platforms, and license →

Comparison at a Glance

Software License Desktop Mobile apps Self-hostable Latest version
Jan Apache-2.0 Windows, macOS, Linux No — 0.8.8
Ollama MIT Linux, Windows, macOS No Yes 0.40.14
LibreChat MIT — No Yes 0.8.8
AnythingLLM MIT Windows, macOS, Linux No Yes 1.19.0
llama.cpp MIT Linux, Windows, macOS No Yes b12103

Versions are the newest stable release tag in each project's repository, checked November 4, 2026.

How We Picked

Picks are ordered by how well each tool serves most people who want a private AI assistant, weighing ease of setup, active maintenance, and platform coverage; every pick released an update within the last month and runs on Linux, Windows, and macOS. Open WebUI is a popular front end for Ollama, but it isn’t included because its license adds a branding restriction that isn’t OSI-approved; see the Open WebUI page. GPT4All isn’t included because it hasn’t published a release since February 2025.

How we evaluate software →

Frequently Asked Questions

What is the best open source AI assistant?

For most people, Jan: it’s a free desktop app that runs models locally with no setup beyond choosing a model. If you want a local model available to many apps, install Ollama.

Can I run a local LLM on Linux?

Yes. Every pick here runs on Linux. Ollama installs with a single script and uses NVIDIA or AMD GPUs when available, Jan is available as a .deb or AppImage, and llama.cpp builds on almost any Linux system.

What hardware do I need to run an AI model locally?

Small models of a few billion parameters run on a modern laptop with 8–16 GB of RAM. Larger models need a GPU with plenty of video memory, or an Apple Silicon Mac with a lot of unified memory. Quantized models, such as the GGUF files llama.cpp uses, cut memory needs considerably.

Are local models as good as ChatGPT or Claude?

Not the largest ones. Open-weight models you can run locally handle writing, summarizing, and everyday questions well, but they generally trail the biggest cloud models on complex reasoning. Several apps here let you use both.