The Architecture of Transcoding: NVENC vs. QuickSync & The AV1 Future (2026)

You've seen, read or heard a lot about transcoding since you've begun your streaming media idealization no doubt. Let's break it down shall we? I've been utilizing home media / streaming servers since roughly 2011 and went through XBMC to Android streaming services, through to Emby, an older media server type I forgot about and finally landed on Plex in 2013. Although this can be a wildly deep topic, I'll just explain what matters for transcoding when it comes to streaming media servers like Plex/Jellyfin.

Why Transcode?

• Different Devices, Different Needs: Your streaming client (Roku, Firestick, Smart TV) isn’t built to play every video file perfectly. High-resolution 4K HDR content needs a powerful CPU or GPU to decode and display smoothly. Your phone, tablet, or older smart TV might not have the processing power to handle that directly as well and needs additional help.
• Bandwidth / Client Performance: If you try to stream a 4K movie directly from your server to a device that can’t handle it, you'll likely experience buffering, stuttering, or a generally poor viewing experience and worst case, it won't work at all.
• Subtitle Burn in: You could have a RTX5090 in there for your transcoding, but the second someone choses to burn in PGS Subtitles - WABAM - CPU single-threaded software burn.

The Direct Play "Myth" (I Only Want Direct Play!)

You think you don't need transcoding because you 'Direct Play' everything at home... Then you travel, try to stream 4K to your iPad on hotel Wi-Fi, and your server crashes.

Going the "No Transcoding Required" is 100% doable if you know exactly what:

• client devices you, your fam or friends will be playing from and;
• WHERE those devices are/will be (mobile? over 5G/LTE? Spotty connections? Trains? Planes?) and;
• you also control the format you store your media in that is compatible across all client devices.

Many guides tell you to "Just Direct Play." This is bad advice and can set you up to have to rebuild not long after!

• Scenario A: You are on LTE/5G. You cannot stream a 120Mbps bitrate file. You need transcoding to squash it to 8Mbps.
• Scenario B: You are using a web browser (Firefox/Chrome). They often don't support HEVC (H.265). Your server must transcode to H.264.

The Rule: Build for Transcoding, hope for Direct Play; not worry about it!

Below you'll see how to build a server that streams 4K Remuxes to even a toaster.

If you'd rather jump straight into streaming and not even worrying about transcoding, or managing your own server, there are multiple ways with Stremio and Real Debrid, or Plex/Jellyfin and Real Debrid!

The Codec Wars (Technical Efficiency)

Here's a breakdown of popular video codecs, their impact & use cases. You should be aiming eventually for full AV1 encode & decode capabilities. This is where everything is going but for now, h.264 is still the main type, with h.265 catching up REALLY fast.

• H.264: Fast, compatible with nearly anything, maybe even your smart fridge, but bandwidth-heavy.
• H.265 (HEVC): Efficient but license-heavy. Not all devices support H.265 due to the licensing.
• AV1: The new king. Accessible on newer hardware but with anywhere from a 20-40% bandwidth savings! This is massive as this means your stream is smaller and less likely to freeze or hangup when you're using hotel wi-fi or on a train etc.

Foundational Transcoding Data Flow

flowchart LR
    %% ==========================
    %% User Devices
    subgraph LANClients ["LAN Clients"]
        LaptopLAN["Laptop (LAN)"]
        PhoneLAN["Phone (LAN)"]
        TabletLAN["Tablet (LAN)"]
    end

    subgraph WANClients ["WAN Clients"]
        LaptopWAN["Laptop (WAN)"]
        PhoneWAN["Phone (WAN)"]
        TabletWAN["Tablet (WAN)"]
    end

    %% Media Server
    subgraph Server ["Plex / Jellyfin Media Server"]
        PMS["Media Server App"]
        Transcoder["Transcoder"]
        HWAccel["Hardware Acceleration"]
        CPUFallback["CPU Fallback"]
        Storage["Media Storage"]
    end

    %% Connections: Requests
    LaptopLAN --> PMS
    PhoneLAN --> PMS
    TabletLAN --> PMS

    LaptopWAN --> PMS
    PhoneWAN --> PMS
    TabletWAN --> PMS

    %% Direct Play Path (LAN preferred)
    Storage -->|Direct Play HLS/DASH/MP4| LaptopLAN
    Storage -->|Direct Play HLS/DASH/MP4| PhoneLAN
    Storage -->|Direct Play HLS/DASH/MP4| TabletLAN

    %% Transcoding Path (WAN or unsupported formats)
    PMS --> Transcoder
    Transcoder --> HWAccel
    Transcoder --> CPUFallback

    HWAccel -->|Transcoded Stream + ABR| LaptopWAN
    HWAccel -->|Transcoded Stream + ABR| PhoneWAN
    HWAccel -->|Transcoded Stream + ABR| TabletWAN
    CPUFallback -->|Transcoded Stream + ABR| LaptopWAN
    CPUFallback -->|Transcoded Stream + ABR| PhoneWAN
    CPUFallback -->|Transcoded Stream + ABR| TabletWAN

    %% Storage connection
    PMS --> Storage

    %% Node Styling
    style LaptopLAN fill:#f87171,stroke:#ff4d4d,stroke-width:2px,color:#111
    style PhoneLAN fill:#fbbf24,stroke:#b45309,stroke-width:2px,color:#111
    style TabletLAN fill:#60a5fa,stroke:#3b82f6,stroke-width:2px,color:#111

    style LaptopWAN fill:#f87171,stroke:#ff4d4d,stroke-width:2px,color:#111
    style PhoneWAN fill:#fbbf24,stroke:#b45309,stroke-width:2px,color:#111
    style TabletWAN fill:#60a5fa,stroke:#3b82f6,stroke-width:2px,color:#111

    style PMS fill:#d8b4fe,stroke:#9333ea,stroke-width:3px,color:#111
    style Transcoder fill:#f472b6,stroke:#be185d,stroke-width:3px,color:#111
    style HWAccel fill:#c084fc,stroke:#7c3aed,stroke-width:2px,color:#111
    style CPUFallback fill:#a3e635,stroke:#4d7c0f,stroke-width:2px,color:#111
    style Storage fill:#34d399,stroke:#059669,stroke-width:2px,color:#111

    %% Subgraph Styling
    style LANClients fill:transparent,stroke:#888,stroke-width:1px,stroke-dasharray:4 4
    style WANClients fill:transparent,stroke:#888,stroke-width:1px,stroke-dasharray:4 4
    style Server fill:transparent,stroke:#888,stroke-width:1px,stroke-dasharray:4 4

Simplified Transcoding Diagram

flowchart TD
    %% ===========================
    %% MEDIA TRANSCODING PIPELINE
    %% ===========================

    subgraph Client["👤 Client Device"]
        PLAY["📺 Playback Request
Resolution / Codec / HDR Info"]
    end

    subgraph Server["🖥️ Plex / Jellyfin Media Server"]
        DEC1["🔐 Decrypt
DRM / Container Encryption"]
        DEMUX["📦 Demux
Split Video / Audio / Subs"]

        DECODE["🎞️ Decode (VRAM Eater)
H.264 / H.265 / AV1
GPU Video Decoder"]

        TONEMAP["🌈 Tone Map (Compute Heavyweight)
HDR ➜ SDR
Color Space Conversion"]

        SCALE["📐 Scale
4K ➜ 1080p / 720p"]

        ENCODE["🎥 Encode (Quality Bottleneck)
NVENC / Quick Sync / VCN"]

        MUX["🧩 Mux
Repackage Stream"]
    end

    subgraph Output["🌐 Stream Delivery"]
        STREAM["📡 Adaptive Bitrate Stream
HTTP / DASH / HLS"]
    end

    PLAY --> DEC1
    DEC1 --> DEMUX
    DEMUX --> DECODE
    DECODE --> TONEMAP
    TONEMAP --> SCALE
    SCALE --> ENCODE
    ENCODE --> MUX
    MUX --> STREAM

    %% ===========================
    %% STYLING
    %% ===========================

    style Client fill:transparent,stroke:#888,stroke-width:1px,stroke-dasharray:4 4
    style Server fill:transparent,stroke:#888,stroke-width:1px,stroke-dasharray:4 4
    style Output fill:transparent,stroke:#888,stroke-width:1px,stroke-dasharray:4 4

    style PLAY fill:#93c5fd,stroke:#1e3a8a,stroke-width:2px,color:#111

    style DEC1 fill:#e5e7eb,stroke:#374151,stroke-width:2px,color:#111
    style DEMUX fill:#e5e7eb,stroke:#374151,stroke-width:2px,color:#111

    style DECODE fill:#fecaca,stroke:#b91c1c,stroke-width:3px,color:#7f1d1d
    style TONEMAP fill:#fde68a,stroke:#ca8a04,stroke-width:3px,color:#7c5800
    style SCALE fill:#bbf7d0,stroke:#15803d,stroke-width:2px,color:#14532d
    style ENCODE fill:#ddd6fe,stroke:#6d28d9,stroke-width:3px,color:#3b0764

    style MUX fill:#e5e7eb,stroke:#374151,stroke-width:2px,color:#111
    style STREAM fill:#60a5fa,stroke:#1d4ed8,stroke-width:2px,color:#111

Simplified Transcoding Decision Matrix

flowchart TD

%% =========================
%% START
%% =========================
A["🎬 Client Requests Video"]

%% =========================
%% CHECK CLIENT SUPPORT
%% =========================
B{"📺 Client Supports
Video Codec?"}

%% =========================
%% CHECK CONTAINER
%% =========================
C{"📦 Container Supported?"}

%% =========================
%% CHECK BITRATE / NETWORK
%% =========================
D{"🌐 Network Fast Enough?"}

%% =========================
%% RESULTS
%% =========================
E["✅ Direct Play
No Server Processing"]

F["🔄 Direct Stream
Container Remux"]

G["🔥 Full Transcode
GPU / CPU Processing"]

%% =========================
%% FLOW
%% =========================
A --> B

B -- Yes --> C
B -- No --> G

C -- Yes --> D
C -- No --> F

D -- Yes --> E
D -- No --> G

%% =========================
%% STYLING
%% =========================
classDef start fill:#ede7f6,stroke:#512da8,stroke-width:2px,color:#311b92;
classDef decision fill:#e3f2fd,stroke:#1e88e5,stroke-width:2px,color:#0d47a1;
classDef success fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px,color:#1b5e20;
classDef remux fill:#fff3e0,stroke:#ef6c00,stroke-width:2px,color:#e65100;
classDef transcode fill:#ffebee,stroke:#c62828,stroke-width:3px,color:#b71c1c;

class A start;
class B,C,D decision;
class E success;
class F remux;
class G transcode;

Hardware Architecture: ASIC vs. CUDA/EU

Not all hardware is equal, that we all understand. But why doesn't an RTX4090 transcode much faster than a 4060 for example? Or why is an old GTX1650 such a transcoding beast, yet reddit says I should upgrade/I'm missing out? There's some nuances to understand.

• NVIDIA (NVENC): NVIDIA's magic is simply a dedicated ASIC (Application Specific Integrated Circuit) chip on their cards. It doesn't touch the CUDA cores (mostly).
  • Mythbusting:
    • Generally it's the exact same ASIC across a generation of GPUs, this is why a RTX4090 barely outperforms a 4060 at transcoding. It's using the same ASIC chip!
    • Old trusty GTX1650/1660 can transcode ~20+ 720/1080p streams, and even a pile of 4k streams, but you cannot however encode (create/transcode) AV1, only decode (play!). You also lose out on HDR tonemapping and better colour with newer generation cards such as RTX3000 series and higher cards.
• Intel (QuickSync): Same idea as an ASIC, but more highly locked down. Basically Intel's QuickSync is a hardware specialized for one single function called "fixed-function". It does this one thing, very efficiently which is video encode/decode and that's it.
• The "Unlock" Hack: NVIDIA infamously places an artificial limit on their consumer grade GPUs, limiting their simultaneous number of video encode/decodes. I believe it's currently set to 4.
  • Follow this link for the hack which easily bypasses this limit.

Transcoding Hardware Comparisons

Let's do a bit of a deep-dive into how Intel QuickSync compares to an RTX3060 to give you an idea of why some chose a GPU, or if an Intel chip will be good enough for your use case. We'll compare in a few different metrics to really give you an idea on what to chose...

📊1. The 2026 Hardware Showdown (Concurrent Streams)

Notes:

• RTX 3060's NVENC excels at H.264 and HEVC encoding with better concurrency and efficiency.
• Intel Arc (or UHD 770 to a lesser extent) gains an edge if you want native AV1 encoding, but loses ground on sheer concurrent streams vs the 3060.

🎛️ 2. Codec / Feature Support Comparison

🎨 3. Quality (Bit Depth, Colour, HDR Support, Encoding Artifacts)

✅ Pros / Cons Summary (for Plex / Jellyfin / Handbrake / FFmpeg / Emby style workloads)

🚩 Transcoding Summary:

• For a heavy Plex/Jellyfin server: The RTX 3060 is still miles ahead in real-world transcoding performance and stability, especially for HDR tone-mapping and concurrent streams. Even an RTX 3050 would suffice.
• If you’re focused on energy use, low volume transcodes, or want AV1 encoding: The Intel Arc A380 or similar iGPU generation could make sense.
• Plex's software stack today (as of mid-2025) still favours NVIDIA NVENC for stability, tone-mapping, and overall encoding quality, with Quicksync being more of a budget or niche (simplicity) option.
• For complete hardware specs and recommendations head over to the Core Lab Homelab Hardware Guide!

The Hardware "Tiers" List

What about transcoding affordably? Absolutely! You can use whatever CPU you already own, fire up your streaming media server of choice, and start watching. I stared with a CPU only and used that for quite a few years. I was also using a GTX 1650 (non-ti) for many years! You don't need to break the bank. Let's breakout some tiers you can aim for though...

Tier F (Free!) - Cost $0

Tier S (The Efficiency King): Intel N150 / N97 (Mini PCs) - Low-to-Medium Cost

Tier S Hardware Recommendations - WHY?
For $150, this chips sips 6W of power and handles 4x 4K Tone Mapped streams. It is the default choice for 99% of users.

For a low-to-medium dedicated device, these KAMRUI are the affordable option (Post DRAM-pocalypse Jan 2 2026) on these Mini-PCs: AK1PLUS Mini PC

Field reports show this can reasonably pump out about 3-5 4k to 1080p streams. Field research example here and here and a great deep dive here.

Main advantage of this mini-PC right now, CHEAP! Last April there was a massive sale, maybe this April there will be again!

Better overall, still cheap
Grab the newer version ACEMAGIC V1 Mini PC Twin Lake N150, but about $50 more.

Many many more options and price ranges here, but make sure to do your homework and ensure it'll suit your needs. Such as having a NIC or two!

My dedicated hardware page has my latest & greatest recommendations, updated monthly.

Tier A (The Specialist): Intel Arc A310 / A380 - Medium-to-High Cost

Tier A Hardware Recommendations - WHY?

Forget what you read in 2024. With Kernel 6.8+, Arc is now stable on Jellyfin and supports AV1 Encode for roughly $160 CAD. It is the best upgrade for old servers/systems and breathes new life into your setup!

Check the price here.

Tier B (The Brute Force): NVIDIA (RTX 3060 / 4060)

Updated (Jan 2 2026) High End Home Server Build

I didn't include a GPU recommendation in this build b/c of the Intel i5-13600k. This chip's Quicksync can get you about 20x 1080p streams and/or 4-5 4k streams! Very powerful for no extra monies!

If you're looking to make use of a GPU you already own, or do more streams then you would need to utilize a GPU. I recommend Intel Arc A380 or, RTX 3060 12GB believe it or not. Before you've read the charts above, this would seem ridiculous but I am sure now you realize the correlations here.

Before you read on, checkout Core Lab's Hardware recommendations which include a ton of detailed info and have many pre-designed & specialized builds. You can also checkout my custom server/NAS.

Now you have the hardware pipeline built. But a fast GPU is useless if your Plex Database is fragmented or your 'Temporary Transcode Directory' is thrashing your SSD.

Here is how you optimize Plex to tune your buffers, RAM disks, and database settings for maximum performance.

Resource Links:

1. Plex's guide on hardware transcoding (They put a LOT of work into re-working this entire section and guide of their site...)
2. Plex's own google sheets comparison of NAS that can do H/W transcoding, quite exhaustive! If you go the pre-purchased route, here's where you should definitely cross-check...
3. Plex official docker image with prebuilt docker-compose setups for various networking & hardware transcoding support!
4. Visit our Ultimate Self-Hosted Media Guides!

🔧 Troubleshooting the Pipeline: The Anatomy of a Buffer

When a stream buffers, it is rarely "magic." It is a mathematical certainty caused by a bottleneck in one of three stages: Compute, I/O, or Network.

Use this checklist to diagnose the choke point.

1. The "Subtitle Burn" (The Silent Killer)

The Symptom: You have an RTX 4090, but your anime or 4K Remux is stuttering. The Cause: Image-based subtitles (PGS, VOBSUB, SSA/ASS). Most client devices (Roku, older Android TVs) cannot natively render these subtitle formats. Plex is forced to "burn" them into the video frames on the fly.

• The Bottleneck: This process is often single-threaded on the CPU. It frequently happens before the frame hits the GPU encoder. Your massive GPU sits idle waiting for a single CPU core to finish painting text onto the frame.
• The Fix: Switch to text-based subtitles (.SRT) to allow "Direct Play," or upgrade to a client that handles PGS natively (NVIDIA Shield Pro / Apple TV).

2. The I/O Thrash (The "Spindle" Bottleneck)

The Symptom: Buffering occurs only when multiple people are watching, or when *arr apps are moving files. The Cause: You are transcoding directly to your mechanical HDD array. Transcoding is a read/write heavy operation. If Plex reads the 80Mbps movie file and writes the temporary chunks to the same mechanical drive, the drive heads begin to thrash (high IO Wait).

• The Fix: Move your "Transcoder Temporary Directory" to a dedicated SSD or, ideally, a RAM Disk (/dev/shm on Linux). (See ourOptimization Guidefor the setup).

3. Bitrate Spikes vs. Average Bitrate

The Symptom: The movie plays fine for 10 minutes, then buffers during an explosion scene. The Cause: You trusted the "Average Bitrate." A "20Mbps" 4K stream is an average. During high-action scenes, the Peak Bitrate can spike to 80Mbps+. If your upload speed (or the hotel Wi-Fi) is capped at 30Mbps, you will drain the client buffer during the explosion.

• The Fix: Force a lower quality (transcode) on the client side (e.g., "Convert to 1080p 10Mbps") to clamp the bitrate ceiling.

4. The Tone Mapping Crush

The Symptom: HDR content buffers, but SDR content plays fine. Colors look washed out. The Cause: Software Tone Mapping. Converting HDR (10-bit) to SDR (8-bit) is mathematically expensive. If you haven't explicitly enabled Hardware Tone Mapping in Plex/Jellyfin, your CPU is trying to do this math in software and failing.

• The Fix: Ensure "Enable HDR Tone Mapping" is checked and that you have the correct drivers installed (OpenCL for AMD/Intel, proprietary drivers for NVIDIA).

FAQ: Architecture & Hardware

• Why does my 4K stream buffer even with a fast GPU? This is often caused by Subtitle Burn-in. If your client device (like a web browser) can't natively display PGS or VOBSUB subtitles, the server must "burn" them into the video frames. This is a single-threaded CPU task that bypasses your GPU, often causing the server to choke regardless of your graphics card.
• What is "Tone Mapping" and why do I need it? If you play a 4K HDR movie on a standard 1080p SDR screen, the colors will look "washed out" or grey. Tone mapping is the mathematical process of converting those HDR colors so they look vibrant on older screens. NVIDIA and Intel QuickSync (on Linux) handle this well; without hardware support, your CPU will likely hit 100% usage immediately.
• Can I use an AMD GPU for transcoding? While possible, AMD's VCE/VCN encoder is generally less supported in Plex and Jellyfin compared to NVIDIA (NVENC) and Intel (QuickSync). For the best "set it and forget it" experience, Intel or NVIDIA is recommended.
• Do I need a Plex Pass for transcoding? Yes. To use Hardware Acceleration (your GPU/iGPU) in Plex, you must have an active Plex Pass subscription. Jellyfin, however, offers hardware transcoding for free.

Q: Do I really need a GPU? My CPU is an i9-14900K.

A: For a pure media server? Maybe not. Modern Intel CPUs (11th Gen+) have the QuickSync media engine built-in. This is effectively a "GPU" dedicated to video. It is distinct from the main CPU cores.

• However: If you run an AMD Ryzen CPU (without an iGPU), you absolutely need a dedicated GPU. A 24-core Threadripper will choke on 4K transcoding because it lacks the fixed-function ASIC logic to decode HEVC efficiently.

Q: Is "RAM Transcoding" dangerous? Will I run out of RAM?

A: No. Transcoding creates small, temporary chunks (usually 1-2 minutes of video). Once the client watches them, they are deleted. Even with 5 concurrent 4K streams, your RAM usage for the transcode folder rarely exceeds 2-3GB.

• Pro Tip: If you use Docker, map /transcode to /tmp or a tmpfs volume to save your SSD from write-wear death.

Q: Why does AV1 matter if I don't have AV1 files?

A: It matters for the future. Right now, you mostly decode HEVC. But soon, you will want to encode to AV1 for remote streaming. AV1 delivers the same visual quality as H.264 at 30-40% lower bitrates. This is a game-changer for users with limited upload speeds (e.g., 20Mbps cable upload).

Q: Can I mix NVIDIA and Intel GPUs?

A: Yes, but pick a lane. Plex generally prefers one device. You can use the Intel iGPU for Plex (QuickSync) and pass the NVIDIA GPU to Tdarr (for background library optimization). Do not try to make them "SLI" a single stream; the PCIe latency cost of moving frames between cards outweighs the benefit.