NVIDIA Toolkit Setup on Linux

Introduction / why?

Now I'm actually a bit of an underdog (AMD) fanboy, but in the end a real tech/nerd uses the best tool available, to do the job. NVIDIA is wearing the King's crown for GPU accelerated functions like video transcoding and especially, AI.

This guide will walk you through installing the NVIDIA Toolkit, drivers, and setting up your docker runtime to make use of it for GPU acceleration of any dockers you so chose, such as Plex / Jellyfin!

Prerequisites

• A supported NVIDIA GPU with a min compute capability of 3.0
• NVIDIA driver already installed & compatible with CUDA Toolkit version
• This guide is for Debian or Ubuntu or their derivitaves like PopOS!
  • Instructions are for headless Linux but will work w/GUI if you run from CLI.

Part 1 - Linux Driver install (Apt way)

Installing NVIDIA drivers is a piece of cake on Debian based systems. We can install right from apt repository or, from direct download. I recommend the apt repo b/c it's simpler and will update for you when a new version is released by NVIDIA.

First, update your apt repository if you have not done so yet. Use your fav text editor like nano to edit source.list found here:

**Backup** your existing sources.lists!
cp /etc/apt/sources.list /etc/apt/sources.list.bak
nano /etc/apt/sources.list

**For Debian 12** - Replace yours with:
deb https://deb.debian.org/debian bullseye main contrib non-free
deb https://deb.debian.org/debian bullseye-updates main contrib non-free
deb https://deb.debian.org/debian-security bullseye-security main contrib non-free

For Debian 13 Trixie:

deb https://deb.debian.org/debian trixie main contrib non-free non-free-firmware
deb https://deb.debian.org/debian trixie-updates main contrib non-free non-free-firmware
deb https://deb.debian.org/debian-security trixie-security main contrib non-free non-free-firmware

Once you've done that, you're /etc/apt/sources.list should look something like:

deb http://deb.debian.org/debian/ bookworm main contrib non-free non-free-firmware
deb http://security.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
deb http://deb.debian.org/debian/ bookworm-updates main contrib non-free non-free-firmware

Now, install the new drivers!

# Do an apt update with your new sources and upgrade everything, so there's nothing waiting or in the way.
sudo apt update && sudo apt upgrade

# Last upgrade step, moves you from a minor upgrade to a major, if there's any.
sudo apt full-upgrade

# Cleanup!

sudo apt --purge autoremove
sudo reboot

# Search apt for NVIDIA drivers if you want.
sudo apt search nvidia-driver
# At the time of writing the latest driver is 575.57.08

# Install headless server drivers.
sudo apt install nvidia-headless-575-server

# Install NVIDIA Utils (gives us nvidia-smi!)
sudo apt install nvidia-utils-575-server

# Search apt for libnvidia-encode. Needed for things like transcoding.
sudo apt search nvidia-encode

# Install libnvidia-encode package.
sudo apt install libnvidia-encode-575-server

After installing all the packages and rebooting (if needed), check if everything is working as it should. Type:

nvidia-smi

Mon Aug 11 09:14:35 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.04             Driver Version: 570.124.04     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3060        Off |   00000000:2E:00.0 Off |                  N/A |
| 39%   44C    P0             45W /  170W |     405MiB /  12288MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A         1580836      C   frigate.detector.tensorrt               266MiB |
|    0   N/A  N/A         3645798      C   /usr/lib/ffmpeg/7.0/bin/ffmpeg          125MiB |
+-----------------------------------------------------------------------------------------+

Part 2 - Installing NVIDIA container toolkit

Now that we're here, we've got 1 more package repo to add/fixup. Copy this whole code block and paste it into your CLI.

# Add the gpgpkey and repository
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg   && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list |     sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' |     sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

If that took, you should get a blank prompt but no errors.

Install the Toolkit, finally

# Update and install NVIDIA Container Toolkit
sudo apt-get update

sudo apt-get install -y nvidia-container-toolkit

# Configure NVIDIA Container Toolkit
sudo nvidia-ctk runtime configure --runtime=docker

sudo systemctl restart docker

Step 3 - Testing / Verification

Now check that everything is running properly, test it out and ensure everything is running properly. We have 2 ways to do this. One is just checking docker itself and another is more intensive, downloading and running a test docker from NVIDIA and actually having it utilize the GPU.

Docker Runtime check.

docker info (sudo docker info if you didn't setup a user properly!)

You should see something like this, look for the 'Runtimes:' line (Bolded). You're looking to see it says nvidia in there.

Running a docker with GPU acceleration

This is as easy as typing:

docker run --gpus all nvidia/cuda:13.0.0-base-ubuntu24.04 nvidia-smi

If that doesn't work due to drivers / type of GPU you have let's try and older one:

docker run --gpus all nvidia/cuda:11.5.2-base-ubuntu20.04 nvidia-smi

Cleanup Time: Warning this is a "Deep clean!"

docker system prune -a
Hit "y" so it removes any extra images/docker cruft from us playing.

Now you're ready to utilize that GPU to accelerate everything and anything you'd like to do in your self-hosted lab, like using it for Plex, or running your very own large language AI model at home!