author Avator

hero Image Blog

DIY AI Chatbot: Free Local Setup

Written by

Author avatar

Abdul Rafay

Share this blog post:

WhatsAppTwitterFacebookLinkedIn

With the rise of large language models (LLMs), it’s no surprise that many are paying between $20 to $50 for AI chatbot subscriptions. However, these services often come with limitations—like daily usage caps—and for the price, the value might not always feel justified. Sure, they have their use on mobile or desktop, but why pay when you can run these models for free on your own hardware?

Don’t worry! I’m here to provide you with a free solution so you won’t have to spend a single dime and can enjoy AI chatbot capabilities for free, right from your own machine.

Requirements

Before we dive in, let’s cover some basic requirements. Don’t worry—these aren’t set in stone, but having them will significantly improve your experience.

  1. Nvidia GPU
    • If you don’t have an Nvidia GPU, no problem. This process can run on CPUs as well, although a GPU will speed things up.
  2. A Processor with More Than 4 Cores
    • Ideally, you’ll have a modern CPU with at least 6 cores for a smoother experience.

Installing Necessary Programs

To get these LLMs running on your computer, you need three main programs:

  1. Docker
  2. Ollama
  3. Ollama WebUI

I’ll guide you through the installation process step by step.

Installing Docker

Installing Docker on Windows

  1. Download Docker Desktop: Head over to Docker’s official site to download the installer.
  2. System Requirements: Windows 11 is required.
  3. Install Docker Desktop: Run the installer and follow the setup prompts.
  4. Configure Settings: After installation, configure either WSL 2 or Hyper-V as per your needs.
  5. Verify Installation: Open PowerShell and run:
docker --version

Installing Docker on Linux

For Linux users, here’s how to install Docker on Debian-based and Arch-based systems.

Installing Docker on Debian

  1. Update and Install Dependencies:
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg lsb-release
  1. Add Docker GPG Key:
curl -fsSL <https://download.docker.com/linux/debian/gpg> | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
  1. Set Up the Stable Repository:
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] <https://download.docker.com/linux/debian> $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
  1. Install Docker:
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

Installing Docker on Arch Linux

  1. Update the Package Database:
sudo pacman -Syu
  1. Install Docker:
sudo pacman -S docker
  1. Enable and Start Docker:
sudo systemctl enable docker
sudo systemctl start docker
  1. (Optional) Add User to Docker Group:
sudo usermod -aG docker $USER

Installing Ollama

For Windows:

Installing Ollama on Windows is a bit trickier but completely doable. You can follow the instructions from their official website.

For Linux:

If you’re using an Nvidia GPU, Ollama can take full advantage of it for GPU acceleration within Docker containers. Follow these steps to get it set up.

CPU-only Setup:

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Nvidia GPU Setup:

  1. Install Nvidia container toolkit.
  2. Run Ollama with GPU support:
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Installing Web UI

To get a user-friendly experience similar to ChatGPT, you can install a Web UI using the following project: Open WebUI.

To install, simply run:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

This will download the WebUI image and start it in Docker. Once the setup is complete, you can access the UI by navigating to localhost:3000 in your browser.

Installing a Language Model

Now, let’s install a language model available on Ollama. Visit their model library and choose the one you prefer. Once selected, run the following command:

ollama run <modelNameAndParameter>

Example:

To install the Llama 3.2 model with 1B parameters, run:

ollama run llama3.2:1b

Remember, the more parameters a model has, the more capable it will be—but also more resource-intensive. Make sure your system can handle it.

Checking the Setup

Once everything is up and running, open your browser and go to localhost:3000. You’ll see the WebUI, similar to ChatGPT’s interface, and you can start using your local models!

Important Notes

These models are quite large and will require a decent amount of storage space and a fast internet connection for downloading. Make sure your setup meets these needs to ensure the best experience.

Conclusion

Running your own AI chatbot for free is not only possible but also relatively simple once you set up Docker, Ollama, and the WebUI. While the models can be large and resource-intensive, they give you a powerful alternative to paid services like ChatGPT without daily limits or costs. Plus, you have complete control over the system.

Why pay for limited access when you can unlock the full power of AI for free? Follow these steps, and you’ll be ready to explore the world of large language models without spending a penny!

Until then, peace out, nerds. ❤️

Advertisements

Comments