Best Open Source AI Assistants
Advertisement
Running an AI assistant on your own hardware has gone from a research project to something you can set up in minutes. Open-weight models from several labs, including Meta’s Llama, Google’s Gemma, Alibaba’s Qwen, and Mistral, can be downloaded and run locally, and a set of mature open source tools handles the hard parts: fetching models, using your GPU, and giving you a chat interface or an API. The result is an assistant whose prompts and documents never leave your machine, with no subscription and no usage limits.
The tools below fit together more than they compete. llama.cpp is the inference engine underneath much of the ecosystem, and Ollama wraps local models in a simple command line and API. Jan and AnythingLLM are complete apps for chatting and for working with your documents, and LibreChat turns a server into a shared, multi-user chat service. How capable a local assistant feels depends mostly on your hardware and the size of the model you can run.
Advertisement
Our 5 Picks
-
Jan
Best for Most people: a desktop chat app for local models
Jan is a free, Apache-2.0 desktop app for chatting with AI models that run on your own computer, with optional connections to cloud providers.
-
Ollama
Best for Running local models for the command line and other apps
Ollama is a free, MIT-licensed tool for downloading and running open-weight LLMs locally, with a command line and an API other apps connect to.
-
LibreChat
Best for A self-hosted, multi-user chat service for teams
LibreChat is a free, MIT-licensed, self-hosted AI chat app with multi-user accounts, agents, and support for Claude, OpenAI, Google, and local models.
-
AnythingLLM
Best for Chatting with your own documents
AnythingLLM is a free, MIT-licensed AI app for chatting with your documents using local or cloud models, as a desktop app or a self-hosted server.
-
llama.cpp
Best for Maximum control and performance for technical users
llama.cpp is an MIT-licensed C/C++ engine for running LLMs locally on CPUs and GPUs, with a command-line chat tool and an OpenAI-compatible server.
Comparison at a Glance
| Software | License | Desktop | Mobile apps | Self-hostable | Latest version |
|---|---|---|---|---|---|
| Jan | Apache-2.0 | Windows, macOS, Linux | No | — | 0.8.8 |
| Ollama | MIT | Linux, Windows, macOS | No | Yes | 0.40.14 |
| LibreChat | MIT | — | No | Yes | 0.8.8 |
| AnythingLLM | MIT | Windows, macOS, Linux | No | Yes | 1.19.0 |
| llama.cpp | MIT | Linux, Windows, macOS | No | Yes | b12103 |
Versions are the newest stable release tag in each project's repository, checked November 4, 2026.
How We Picked
Picks are ordered by how well each tool serves most people who want a private AI assistant, weighing ease of setup, active maintenance, and platform coverage; every pick released an update within the last month and runs on Linux, Windows, and macOS. Open WebUI is a popular front end for Ollama, but it isn’t included because its license adds a branding restriction that isn’t OSI-approved; see the Open WebUI page. GPT4All isn’t included because it hasn’t published a release since February 2025.
Frequently Asked Questions
What is the best open source AI assistant?
For most people, Jan: it’s a free desktop app that runs models locally with no setup beyond choosing a model. If you want a local model available to many apps, install Ollama.
Can I run a local LLM on Linux?
Yes. Every pick here runs on Linux. Ollama installs with a single script and uses NVIDIA or AMD GPUs when available, Jan is available as a .deb or AppImage, and llama.cpp builds on almost any Linux system.
What hardware do I need to run an AI model locally?
Small models of a few billion parameters run on a modern laptop with 8–16 GB of RAM. Larger models need a GPU with plenty of video memory, or an Apple Silicon Mac with a lot of unified memory. Quantized models, such as the GGUF files llama.cpp uses, cut memory needs considerably.
Are local models as good as ChatGPT or Claude?
Not the largest ones. Open-weight models you can run locally handle writing, summarizing, and everyday questions well, but they generally trail the biggest cloud models on complex reasoning. Several apps here let you use both.