Network & Transit 3 min read Last updated: 2026-10-23

MTU Probing, TCP MSS Clamping & Path MTU Discovery (PMTUD) Troubleshooting

Diagnose and eliminate cross-border PMTUD blackholes where ping succeeds but HTTPS, SSH, or WireGuard tunnels hang, using ICMP DF probing, tcp_mtu_probing, and iptables MSS clamping.

One of the most perplexing networking symptoms on international links or encrypted overlay tunnels (WireGuard, IPsec, GRE, VXLAN) is the PMTUD (Path MTU Discovery) Blackhole: ICMP ping works with 0% loss, initial TCP three-way handshakes succeed, and small SSH commands respond instantly—yet running htop, git clone, or loading an HTTPS certificate hangs indefinitely. This guide explains how MTU & TCP MSS work and how to permanently resolve fragmentation blackholes.

1. Understanding MTU, TCP MSS, and PMTUD Blackholes

  • MTU (Maximum Transmission Unit): The largest Layer-3 IP packet (in bytes) that a network interface can transmit without fragmentation. Standard Ethernet defaults to 1500 bytes.
  • MSS (Maximum Segment Size): The maximum TCP payload inside a single IP packet, calculated as MTU - 20 bytes (IPv4 Header) - 20 bytes (TCP Header). For a 1500 MTU link, standard IPv4 MSS = 1460 bytes (or 1440 bytes over IPv6 due to the 40-byte IPv6 header).

Common Network MTU & MSS Reference Table:

Link / Tunnel Type Encapsulation Overhead Optimal Interface MTU Optimal IPv4 TCP MSS Optimal IPv6 TCP MSS
Standard Datacenter Ethernet 0 Bytes 1500 1460 1440
Residential PPPoE Broadband 8 Bytes 1492 1452 1432
WireGuard over IPv4 60 Bytes 1420 1380 1360
WireGuard over IPv6 80 Bytes 1400 1360 1340

[!WARNING] Why Do Connections Hang? Modern TCP packets set the DF (Don’t Fragment) bit. When a 1500-byte packet hits a residential PPPoE router (1492 MTU) or tunnel interface (1420 MTU), the router drops the oversized packet and returns an ICMP Fragmentation Needed (Type 3, Code 4) message. If an intermediate firewall drops that ICMP notification, the sender keeps retransmitting 1500-byte packets forever—resulting in a hung connection (PMTUD Blackhole).


2. Probing Exact Path MTU with ping (Don't Fragment)

You can determine the exact maximum unfragmented packet size supported between your server and any target IP by setting the DF (Don’t Fragment) flag.

Note that ping -s specifies the ICMP payload size, which excludes the 20-byte IPv4 header and 8-byte ICMP header (28 bytes total overhead):

# Test if standard 1500 MTU (1472 payload + 28 header = 1500) passes cleanly
ping -M do -s 1472 -c 4 target_ip

# Test if PPPoE 1492 MTU (1464 payload + 28 header = 1492) passes cleanly
ping -M do -s 1464 -c 4 target_ip

If ping -M do -s 1472 returns message too long, mtu=1492 or 100% packet loss, while -s 1464 or -s 1392 succeeds immediately, you have confirmed a Path MTU bottleneck along the route.

For IPv6 (where the header overhead is 40 bytes IPv6 + 8 bytes ICMPv6 = 48 bytes):

# Test 1500 MTU over IPv6 (1452 + 48 = 1500)
ping6 -M do -s 1452 -c 4 2001:4860:4860::8888

3. Solution 1: Enable Kernel RFC 4821 Smart MTU Probing

The Linux kernel includes built-in Packetization Layer Path MTU Discovery (PLPMTUD, RFC 4821). When enabled, if the kernel detects repeated TCP retransmissions indicative of an ICMP blackhole, it automatically reduces the TCP segment size on that connection without relying on external ICMP packets:

cat << 'EOF' | sudo tee /etc/sysctl.d/98-pmtud.conf
# Enable smart TCP Path MTU Probing when blackholes are detected (1 = on blackhole, 2 = always)
net.ipv4.tcp_mtu_probing = 1

# Set safe fallback base MSS (1024 bytes) for blackhole recovery
net.ipv4.tcp_base_mss = 1024
EOF

sudo sysctl --system

4. Solution 2: Automatic TCP MSS Clamping (iptables / nftables)

If your FlawlessNode cloud server operates as a WireGuard / IPsec gateway, router, or container host, applying TCP MSS Clamping on the mangle table automatically rewrites the MSS field in TCP SYN handshake packets to match the outgoing interface’s Path MTU:

4.1 Applying MSS Clamping via iptables (IPv4 & IPv6)

# Automatically clamp TCP MSS to Path MTU for forwarded and outbound traffic
sudo iptables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
sudo ip6tables -t mangle -A POSTROUTING -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

4.2 Applying MSS Clamping via nftables

sudo nft add table inet mangle
sudo nft 'add chain inet mangle postrouting { type filter hook postrouting priority mangle; policy accept; }'
sudo nft 'add rule inet mangle postrouting tcp flags syn / syn,rst tcp option maxseg size set rt mtu'

4.3 Setting Explicit MTU in WireGuard Configurations

In /etc/wireguard/wg0.conf, always specify an explicit MTU = 1420 (or 1400 if transporting over IPv6 or PPPoE):

[Interface]
PrivateKey = <server_private_key>
Address = 10.66.66.1/24
ListenPort = 51820
MTU = 1420

[!TIP] Combining MTU = 1420 on tunnel interfaces with net.ipv4.tcp_mtu_probing = 1 in sysctl eliminates 99.9% of cross-Pacific TLS handshake stalls and large-packet drops.

Was this page helpful?