PMTU Blackhole over WireGuard (NNTP server hang)

Symptom

An NNTP server (pnntprss, user systemd unit, port 4321) reachable only via a WireGuard tunnel “hangs forever” for clients on the far side:

Root cause

MTU blackhole (PMTU blackhole).

It is asymmetric: each side encapsulates independently, so one direction can work while the other dies. Here server→client stalled (NNTP responses), while client→server small packets were fine.

Why it is hard to find

  1. Small packets always work — banner, commands, ACKs are all < 1408 bytes. The connection establishes and looks healthy, then hangs on the first bulk transfer. Looks like an application bug, not a network bug.
  2. No ICMP PTB reaches TCP → no PMTU clamping, silent infinite retransmission.
  3. Asymmetric — either direction can break independently (both sides need the fix).
  4. Silent — nothing in logs, no obvious counter.
  5. Affects any tunnel tech: WireGuard (default 1420), OpenVPN, IPsec/VTI, GRE, 6in4/4in6, GTP, Docker bridges.

Detection recipe (what nailed it)

  1. DF-set ping sweep to find the exact inner threshold:

    for s in 1376 1380 1384; do
      ping -c 1 -W 2 -M do -s $s 192.168.1.187
    done
    # inner 1408 (payload 1380): 0% loss
    # inner 1412 (payload 1384): 100% loss   ← threshold
  2. Measure the outer path MTU to the WireGuard endpoint:

    wg show wg0                 # endpoint
    ping -6 -c 1 -W 2 -M do -s 1444 <endpoint>   # outer 1492: OK
    ping -6 -c 1 -W 2 -M do -s 1446 <endpoint>   # outer 1494: Packet too big
  3. Confirm overhead: 1408 inner + 80 (32 WG + 40 IPv6 + 8 UDP) = 1488 ≤ 1492 ✓, 1412 inner → 1492 (borderline, dropped), and the configured 1420 inner → 1500 > 1492 ✗

  4. Capture — oversized UDP leaves the host with nothing coming back:

    sudo tcpdump -ni eth0 "ip6 and udp and host <endpoint>"
  5. Watch the stalled TCP socket retransmit with growing backoff:

    ss -tni dst <client>

Fix

Lower MTU on both tunnel endpoints (each encapsulates independently):

# /etc/wireguard/wg0.conf  (both sides)
[Interface]
MTU = 1380

1380 + 80 = 1460 ≤ 1492 → 32 bytes headroom.

Verification

# inner oversized packet now refused locally instead of blackholed
ping -c 1 -M do -s 1384 <client>   # → "sendmsg: Message too long" (expected)

# fresh connections pick up the new PMTU
ss -tni dst <client>               # → pmtu:1380 mss:1328

# two-way bulk test (SSH banner + complete kexinit inbound, large write outbound)

Result: NNTP client reads news again; SSH kexinit (1040 B) arrives complete; 8000-byte writes fully acked.

Key numbers

Value Meaning
1420 WireGuard default MTU (assumes 1500 path)
1492 Real path MTU → PPPoE uplink (1500 − 8)
1408 Max inner packet that survived before the fix
80 WG outer overhead: 32 WG + 40 IPv6 + 8 UDP
1380 Chosen MTU: 1380 + 80 = 1460 ≤ 1492
1328 Resulting TCP MSS (1380 − 52)

Lesson

When a protocol “hangs” over a tunnel but small packets work, suspect PMTU blackhole first: DF ping sweep for the inner threshold, compare with the outer path MTU, fix both tunnel endpoints.