Getting your homelab online was Day 1. Day 2 is about absolute control.

Most "Pro" home networks stop at basic ad-blocking and simple port forwards. To truly master your infrastructure, you need to dictate exactly how traffic enters your network, how internal devices resolve their identity and time, and how your router handles congestion on high-speed connections.

These techniques are commonly used in advanced homelab firewall setups to maintain security, performance, and full control over self-hosted infrastructure.

This guide walks through advanced OPNsense networking techniques including Cloudflare bypass rules, DNS and NTP interception, VLAN isolation, and SQM traffic shaping.

Why do this? This helps further secure your homelab from things like NTP & DNS Flood amplification attacks (low and "slow" DDoS) whilst allowing you to still host services, securely.

Here is your blueprint for a "God-Mode" OPNsense architecture.

Architecture we're aiming for:

flowchart TD

    A[Internet]
    B[Cloudflare Proxy]
    C[OPNsense Firewall]

    D[SWAG Reverse Proxy]

    E[Internal Applications]
    F[Jellyfin Media Server]

    G[AdGuard DNS]
    H[Local NTP Server]

    A --> B
    B --> C

    C -->|HTTPS Apps| D
    D --> E

    C -->|Direct Media Port
Cloudflare Bypass| D
    D --> F

    C -->|Forced DNS| G
    C -->|Forced NTP| H

    %% Groups
    subgraph Edge
        A
        B
    end

    subgraph Network_Perimeter
        C
    end

    subgraph Application_Layer
        D
        E
        F
    end

    subgraph Infrastructure_Services
        G
        H
    end

    %% CoreLab cyberpunk styling
    classDef edge fill:#000000,stroke:#00ff88,color:#00ff88,stroke-width:2px
    classDef firewall fill:#001a12,stroke:#00ff88,color:#00ff88,stroke-width:3px
    classDef apps fill:#001f1a,stroke:#00ffaa,color:#00ffaa
    classDef infra fill:#001a1a,stroke:#00dddd,color:#00dddd

    class A,B edge
    class C firewall
    class D,E,F apps
    class G,H infra

    %% Clickable documentation links
    click C "https://corelab.tech/opnsensept1/" "OPNsense configuration guides"
    click D "https://corelab.tech/swag-reverse-proxy-guide/" "SWAG reverse proxy setup"
    click F "https://corelab.tech/ultimate-media-server-guide/" "Media server architecture"
    click G "https://corelab.tech/cloudflare-protection-free/" "DNS services"
    

Phase 1: How to Bypass Cloudflare for Jellyfin or Plex in OPNsense

If you are routing your web traffic through Cloudflare tunnels or their proxy (the "Orange Cloud"☁️) and restricting your OPNsense WAN to only accept traffic from Cloudflare IPs, you have a highly secure setup.

When self-hosting behind Cloudflare, many homelab users eventually need to bypass the proxy for specific services such as Jellyfin or Plex. Cloudflare's free tier has strict Terms of Service against heavy video streaming, and their proxy only supports specific ports (80, 443, etc.). If you want to run Jellyfin or Plex with a domain name, you must bypass Cloudflare, or risk getting your account banned, or suffer buffering. No one wants buffering!

The solution is to create a direct NAT rule in OPNsense that allows traffic to reach your reverse proxy or media server without passing through Cloudflare’s proxy network. This preserves the security benefits of Cloudflare for normal web traffic while allowing high-bandwidth media services to connect directly.

Here is how to poke a precise, safe hole through your locked-down WAN. You can do this for multiple services, if needed. Here's what this looks like in Cloudflare DNS records.

Simply login, hit your DNS settings and create a new A record. I used jf for mine but you could use jellyfin.yourdomain.com if you so chose etc...

The OPNsense Port Forward Config

You need a NAT Port Forward that intercepts port 8096 and bypasses your strict Cloudflare-only firewall rules.

Network Address Translation — OPNsense documentation

1. Create an Alias (Optional but Recommended): Go to Firewall -> Aliases and create a GeoIP alias allowing only your home country, or an alias for specific remote IP addresses you trust. Here's what my alias looks like:

1. The NAT Port Forward: Go to Firewall -> NAT -> Port Forward and add:
   • Interface: WAN
   • TCP/IP Version: IPv4
   • Protocol: TCP
   • Source: Any (or your custom Alias from Step 1 to restrict who can see the port)
   • Destination: WAN Address
   • Destination Port Range: 8096 to 8096
   • Redirect Target IP: <Your reverse proxy IP or alias>
   • Redirect Target Port: 8096
   • Filter Rule Association: Add associated filter rule.
2. Rule Order is Critical: Go to Firewall -> Rules -> WAN. OPNsense processes rules top-down. You must ensure this new Jellyfin NAT rule sits above your "default deny all" drop rule at the bottom. I have might snuggled directly below my Cloudflare locked down reverse proxy rules:

Result: Web traffic remains fully shielded by Cloudflare, but your heavy media streams bypass the proxy directly to your server, keeping you compliant and buffer-free.

Phase 2: DNS and NTP Interception for Network Control

In a segmented network, "polite" devices use the DNS and NTP servers you provide via DHCP. "Rude" devices (like IoT gear and Smart TVs) often have hardcoded servers like 8.8.8.8 or pool.ntp.org, bypassing your local security, ad-blocking, and logging entirely.

We fix this by building a Transparent Redirect system. We will "lie" to these devices by intercepting their requests and forcing them into our local stack. Some people call this NAT reflection, industry calls if DNS Hijacking 👇

What is DNS Hijacking in OPNsense?

DNS hijacking in OPNsense refers to intercepting outbound DNS requests and redirecting them to a local resolver such as AdGuard Home or Unbound. This prevents devices from bypassing your network policies by using external DNS services like Google DNS (8.8.8.8) or Cloudflare DNS (1.1.1.1).

Many IoT devices and smart TVs hardcode their own DNS servers, which can bypass ad-blocking, logging, and security filtering. By redirecting DNS traffic at the firewall level, OPNsense ensures all devices resolve domains through your internal DNS infrastructure.

1. The Intercept (NAT Hijacking)

When a device tries to talk to an external DNS or NTP server, OPNsense grabs the packet and answers as if it were the intended destination.

NTP Redirect (Force Local Time) Setup:

The NAT Port Forwards: Go to Firewall -> NAT -> Port Forward and add:

• Interface: HomeRZ, IoT
• Destination / Invert: Checked
• Destination: This Firewall
• Redirect Target IP: 127.0.0.1 (Redirects to OPNsense's own NTP service)
• Port: 123 (UDP)

DNS Redirect (Force AdGuard):

• Interface: HomeRZ, IoT
• Destination / Invert: Checked
• Destination: This Firewall
• Redirect Target IP: 10.0.0.110 (Your AdGuard macvlan IP)
• Port: 53 (TCP/UDP)

2. The Hybrid DNS Handshake

We want AdGuard Home's UI for blocking, but OPNsense Unbound's power for resolving local hostnames (like nas.local).

1. OPNsense Unbound: Move Unbound to port 5353 and enable DHCP Registration.

AdGuard Home: In DNS Settings, set your Upstream DNS using conditional syntax:

[/home.arpa/]10.0.0.1:5353
[/168.192.in-addr.arpa/]10.0.0.1:5353
https://dns.cloudflare-dns.com/dns-query

Top lines route local domains to OPNsense. The bottom line sends public queries to an encrypted resolver.

3. Macvlan & The Docker Shim

Running AdGuard on a macvlan network gives it a dedicated IP, but isolates it from the Docker Host. The solution is a bridge shim network. You don't HAVE to do this, but for all my dockers that act as "network appliances" and generally operate properly with their own MAC address (layer 2, broadcast domain etc), I use macvlan. Unifi Network Controller & Adguard use this.

# How to "hot-attach" the shim via CLI to restore host communication
docker network connect --ip 10.0.0.111 omv_shim adguardhome
ip route add 10.0.0.110 omv_shim scope link

4. Floating Rules for NTP Tenant Isolation

If you run a multi-tenant or highly segmented setup, duplicating NTP rules on every VLAN interface gets messy. Use Floating Rules (which evaluate before interface rules when "Quick" is checked) to centralize logic.

Phase 3: Eliminating Bufferbloat with FQ-CoDel SQM

High speed doesn't mean high quality. If your latency spikes when you start a massive download, you have Bufferbloat. We solve this using FQ_CoDel (Fair Queuing Controlled Delay) to ensure "small" packets (gaming, SSH, VoIP) aren't crushed by "big" packets (Steam downloads, 4K streams). Run a Bufferbloat test a few times before you start setting this up, so you have a baseline and can see the difference. Waveform Bufferbloat Test.

Comparison of ISP / Consumer Router vs OPNsense with SQM:

The Three-Layer Shaper Cake:

You'll find these settings in Firewall -> Shaper.

1. Pipes: Create a "Speed Ceiling" at ~90-95% of your actual line speed (e.g., 1350 Mbps for a 1.5Gbps line). This forces the bottleneck into OPNsense where your CPU can manage it, rather than at your ISP's unmanaged node.
2. Queues: Use Source/Destination Masks to ensure bandwidth is shared fairly between all devices on the network.
3. Rules: Set WAN In (Download Queue) and WAN Out (Upload Queue) rules to capture all traffic and sort it into your FQ_CoDel queues.

Bufferbloat Before and After (Results)

Here's the before results picture

Here's the link to that result (Sadness!)

Now here's the result AFTER I configured my SQM & Queues properly:

Here's the link to this result (Less sad!)

Result: Gaming, SSH, and VoIP remain responsive even during large downloads.

Go ahead, make fun of me in the comments below, or better yet, give me some tuning tips 😉

The Hardware Secret: Static Port NAT

OPNsense uses "Asymmetric NAT" by default, which can break game voice chats or Plex remote access. Go to Firewall -> NAT -> Outbound and enable Static Port rules if you have to, for your gaming/media devices. This keeps your internal ports mapped 1:1 to the outside world, ensuring an "Open NAT" status.

The other option (less popular, WAY less secure, but very easy) is to enable UPNP IGD & PCP in your OPNsense. 😱🙀This is your classic convenience / ease of use vs security.

Can read a lot more opinions on it in this Reddit thread.

If you go this route, ensure you select Default Deny in the bottom 1/2 of the setup, and setup the ACLs. Here's how you setup ACL's for Playstation 5:

Gaming Connectivity Reference Table

Pro-Tip:

You can simplify your OPNsense outbound NAT setup, by creating an Alias called Gaming_Devices that contains the static IPs of the consoles and the gaming PC. Then, you only need one Outbound NAT rule at the top of your list to apply the "Static Port" fix to everything in that Alias at once!

Troubleshooting and Verification Tests: The "Trust but Verify" Guide

Once you’ve applied these rules, use these tests to confirm your "God-Mode" network is operational.

• The DNS "Hijack" Test: Open a terminal and run nslookup secret-test.com 8.8.8.8. Check your AdGuard Home Query Log. Even though you asked Google, AdGuard should have intercepted and logged the request.
• The NTP "Force" Test: In OPNsense, go to Firewall -> Diagnostics -> States and search for :123. Look for a device trying to reach an external IP; the NAT column should show it being seamlessly rewritten to 127.0.0.1.
• The Bufferbloat Audit: Run the Waveform Bufferbloat Test. Watch Firewall -> Shaper -> Statistics in real-time. You should see Drops or Marks climb as OPNsense actively manages the queue to maintain your A+ latency grade.

Summary & Next Steps

You're doing fantastic if you're at this stage!

Now you can self-host just about anything, you're really cooking with 🔥

Got a stubborn IoT device that still won't obey your DNS rules? Drop a comment below and let's troubleshoot the logs together!

Another option is to keep securing your stack and take my homelab security checkup test, towards the bottom of my Practical Cybersecurity Roadmap for Homelabs.

The Digital Fortress: A Homelab Security Roadmap

Stop exposing port 32400 to the world. Here is how to build a defense-in-depth strategy for your self-hosted stack. 🛡️

Core LabCore Lab Joe

FAQ

What is an OPNsense Cloudflare bypass rule?

A Cloudflare bypass rule allows specific services such as Jellyfin or Plex to accept direct WAN traffic instead of passing through Cloudflare’s proxy. This is necessary because Cloudflare’s free tier restricts video streaming and non-standard ports.

Why redirect DNS requests on OPNsense?

Redirecting DNS traffic ensures all devices use your internal resolver such as AdGuard Home. Many IoT devices attempt to bypass local DNS settings by hardcoding external resolvers like 8.8.8.8.

What is SQM traffic shaping in OPNsense?

SQM (Smart Queue Management) using algorithms like FQ-CoDel reduces bufferbloat by prioritizing latency-sensitive traffic such as gaming, SSH, and VoIP over large downloads.

Traffic Shaping — OPNsense documentation

Does traffic shaping require powerful hardware?

Yes. Because shaping occurs in software, high-speed connections require stronger CPUs. For gigabit shaping, modern Intel Core, Ryzen, or Xeon-class processors are recommended.

Can OPNsense force devices to use local NTP servers?

Yes. Using NAT redirect rules, OPNsense can intercept NTP requests and force devices to use the firewall’s internal time server instead of public NTP pools.