Networking — DDoS-Protected External IPs via GRE Tunnel¶

Architecture¶

DDoS-protected external IPs from an anti-DDoS provider are routed through a GRE tunnel to the MikroTik router, then into the k3s cluster via MetalLB.

Internet (client)
  ↓
AntiDDoS IP (public)
  ↓  Provider routes to VPS
Transit VPS
  ↓  GRE tunnel (gre-tunnel)
MikroTik router (tunnel endpoint)
  ↓  L2 routing to k3s nodes
k3s cluster nodes
  ↓  MetalLB L2 advertisement
LoadBalancer Services

VPS Configuration¶

GRE Tunnel¶

Created and maintained by a script (gre-update.sh), which runs at boot and every 5 minutes via cron. It resolves the MikroTik's dynamic DNS hostname and updates the tunnel remote if the IP changes.

Routing¶

The AntiDDoS IPs are not locally assigned on the VPS — they must not be, for forwarding to work. Packets arriving at eth0 for the public IPs are forwarded via the GRE tunnel route. MASQUERADE rewrites the source to the tunnel IP so response traffic returns via the GRE tunnel.

# iptables rule — MASQUERADE only, no DNAT
iptables -t nat -A POSTROUTING -o gre-tunnel -j MASQUERADE

Why no DNAT¶

An earlier approach used DNAT to redirect all traffic to the MikroTik tunnel IP. This destroyed the original destination IP, requiring a hardcoded per-IP DNAT rule on MikroTik for each public IP. Removing DNAT fixes this permanently — the original destination is preserved through the GRE tunnel, and MikroTik routes them via L2 to MetalLB without any per-IP rules.

Keepalive¶

The provider's upstream router uses ARP to know which physical host owns the public IPs. Without periodic ARP announcements, the ARP entry ages out and the provider stops routing those IPs to the VPS.

Since the IPs are not locally assigned, standard tools (arping, ping -I) cannot be used. Instead, a Python script uses a raw AF_PACKET socket to craft and send gratuitous ARP frames directly, bypassing the kernel's local address check.

arp-announce.py — sends gratuitous ARP replies (op=2) for each IP on eth0.

ip-keepalive.sh — calls arp-announce.py and logs output. Runs every 5 minutes via cron.

Crontab (root on VPS)¶

*/5 * * * *  /usr/local/bin/gre-update.sh    >> /var/log/gre-update.log    2>&1
*/5 * * * *  /usr/local/bin/ip-keepalive.sh  >> /var/log/ip-keepalive.log  2>&1
@reboot      /etc/network/add-ip.sh
@reboot      /usr/local/bin/gre-update.sh    >> /var/log/gre-update.log    2>&1

MetalLB (k3s cluster)¶

A pool ginernet-pool covering the public IPs is advertised via L2. All client LoadBalancer services use:

annotations:
  metallb.universe.tf/loadBalancerIPs: <public-ip>
  metallb.universe.tf/allow-shared-ip: Primary
  metallb.universe.tf/address-pool: ginernet-pool

Multiple services share one IP on different ports. The allow-shared-ip key must be present on every service sharing an IP.

The kernel uses pool.label as the shared_ip_key annotation value (set in the ip_pools DB table).

Adding a New IP¶

When purchasing an additional AntiDDoS IP:

VPS — ip-keepalive.sh: add the new IP to the arp-announce.py call
VPS — add-ip.sh: add the new IP for boot persistence
VPS — gre-update.sh: add a route for the new IP through the GRE tunnel
MetalLB: edit ginernet-pool IPAddressPool to include the new IP
Database: insert a row into the ip_pools table:

INSERT INTO ip_pools (id, address, port_range_start, port_range_end, label, metallb_pool_name, exposure, enabled)
VALUES (gen_random_uuid(), '<new-ip>', 1024, 65535, 'Primary', 'ginernet-pool', 'MetalLbLoadBalancer', true);

No iptables changes are needed since forwarding is transparent.

Migrating to a New k3s Node¶

When the cluster moves to a new physical node, MikroTik's static routes for the public IPs still point to the old node's IP. Traffic arrives at MikroTik from the GRE tunnel but is forwarded to the wrong host.

Fix: update the static routes in MikroTik's route list to point to the new node's LAN IP (e.g. 172.29.236.105 for c5).

MetalLB's L2 gratuitous ARP announcements will update MikroTik's ARP cache over time, but static routes take precedence and must be updated manually.

Troubleshooting¶

External traffic not reaching services (but reachable from internal VPN)

The break is upstream of the cluster — k3s and MetalLB are not the issue if services are reachable from within the VPN. Check in order:

MikroTik static routes point to the right node?

If the cluster recently migrated to a new node, update the static routes in MikroTik's route list to the new node's LAN IP. This is the most common cause after a migration.

Keepalive ARPs working? (on the Ginernet VPS)

tail -20 /var/log/ip-keepalive.log
# Should show ARP OK every 5 minutes

Public IPs must NOT be on eth0 (would break forwarding):
```
ip addr show eth0 | grep <public-ip>
# Should show nothing
```
If present, run /usr/local/bin/gre-update.sh to remove them.

GRE tunnel up?

ip tunnel show gre-tunnel
ping -c 3 <mikrotik-tunnel-ip>

Traffic arriving at VPS?

tcpdump -i eth0 dst <public-ip> -n
# Then try connecting from a phone (off VPN)

iptables MASQUERADE rule present?
```
iptables -t nat -L -n
# Should show MASQUERADE on gre-tunnel
```
If missing, run gre-update.sh manually to restore.