katran

Multiplexed Multi-Protocol Gatewaying: Bridging TCP, UDP, and QUIC Over a Single Ingress Port Cloud backends, game servers, and edge networks all want fewer exposed ports. Fewer ports mean simpler firewall rules, fewer load balancers to run, and fewer holes in strict corporate or carrier networks. A single public port such as 443 or 8443 can front gRPC and HTTP/2 over TCP, HTTP/3 over QUIC, and a custom real-time UDP protocol. The idea is easy to state and easy to get subtly wrong. This guide covers what the operating system gives you for free, how to classify TCP and UDP traffic safely, where eBPF and XDP help, how QUIC connection IDs affect routing, and what really causes port exhaustion. It ends with a small, tested Go gateway you can build on. --- 1. What “single port” really means A port number is not one namespace. TCP port 443 and UDP port 443 are separate, so a host can listen on both at once. HTTP/3 already works this way: browsers connect over TCP first, then learn about the HTTP/3 endpoint from an Alt-Svc: h3=":443" response header or an HTTPS DNS record (RFC 9460). Because that endpoint is on UDP 443, clients may need to fall back to TCP if a network filters UDP 443. So “one port” means one port number, with three problems to solve on top of it: 1. Transport split. TCP and UDP are delivered to different sockets by the kernel. 2. TCP sub-protocols. TLS (and which application protocol it carries), cleartext HTTP/1.1, cleartext HTTP/2, and custom byte streams all arrive on the same listener. 3. UDP sub-protocols. QUIC and your own datagram protocol share one UDP socket, and UDP gives you no handshake to tell them apart. ┌───────────────────────────────────────────┐ │ Public ingress, port 443 │ └───────────┬─────────────────────┬─────────┘ │ TCP │ UDP ┌─────────▼─────────┐ ┌───────▼────────────────┐ │ Peek: TLS? PRI? │ │ First-byte classifier │ │ HTTP verb? ALPN? │ │ (RFC 9443 ranges) │ └───┬─────┬─────┬───┘ └──────┬──────────┬──────┘ ▼ ▼ ▼ ▼ ▼ gRPC HTTP/1 custom QUIC / custom (h2) TCP HTTP/3 UDP (game) Where the 4-tuple stops being enough When one socket address receives several protocols, the kernel’s lookup (protocol, source address and port, destination address and port) only gets the packet to the right socket. Everything after that is your classifier’s job. | Transport | What you can inspect | Identifying marker | State | | --- | --- | --- | --- | | TCP (TLS, HTTP/2, gRPC) | First bytes of the stream | 0x16 = TLS handshake record; ALPN inside the ClientHello | Kernel connection state | | TCP (cleartext) | First bytes of the stream | PRI * HTTP/2.0… (HTTP/2 preface) or an HTTP method | Kernel connection state | | UDP (QUIC) | First byte of each datagram | High bit set = long header; low-order bits per RFC 9443 ranges | QUIC connection IDs, held by endpoints | | UDP (custom) | Whatever your protocol defines | Your own first-byte convention | Yours to define | --- 2. The socket layer: you don’t need SO_REUSEPORT A common claim is that SO_REUSEPORT is what lets a TCP socket and a UDP socket share a port. It isn’t. Linux tracks TCP and UDP bindings in separate tables, so a TCP listener and a UDP socket can bind 0.0.0.0:443 with no special options. SO_REUSEPORT solves a different problem: letting several sockets of the same protocol bind the same address and port, with the kernel distributing incoming connections or datagrams among them. By default it hashes on the 4-tuple, and all sockets in the group must belong to the same effective user ID. That is how you scale one listener across many threads or processes. In Go, the standard library is enough: ln, _ := net.Listen("tcp", "0.0.0.0:443") // TCP socket pc, _ := net.ListenPacket("udp", "0.0.0.0:443") // UDP socket, same port number Use SO_REUSEPORT when you want per-core listeners or zero-downtime restarts. Section 5 shows how an eBPF program can override its default hashing. --- 3. Classifying UDP: QUIC versus your own protocol This is the hard part. A UDP datagram arrives with no handshake, so the first byte is all you have. What QUIC guarantees (and what it doesn’t) RFC 8999 defines only the properties that stay fixed across every QUIC version: - The most significant bit of the first byte is the header form. 1 means long header, 0 means short header. - Long headers carry a 32-bit Version field (0x00000000 is reserved for Version Negotiation), then length-prefixed Destination and Source Connection IDs. - Short headers carry only the Destination Connection ID, with no length field, so a gateway has to know how long the connection IDs it expects are. The Fixed Bit (0x40) is not an invariant. RFC 8999 lists “QUIC packets always have the second bit of the first byte (0x40) set” among the incorrect assumptions that can be false for a given version. In QUIC version 1 the bit is set, but RFC 9287 lets endpoints that negotiate the grease_quic_bit transport parameter randomize it. Likewise, QUIC version 1 requires a minimum size for client Initial datagrams (1200 bytes, RFC 9000 section 14.1), but that is also a version-1 behavior, not an invariant. A short-header packet has no meaningful “minimum 3 bytes” rule you can build a classifier on. The standard multiplexing scheme: RFC 9443 RFC 9443 (which updates RFC 7983) defines how to demultiplex STUN, ZRTP, DTLS, TURN channels, RTP/RTCP, and QUIC on one UDP socket by looking at the first byte: | First byte | Protocol | | --- | --- | | 0–3 | STUN | | 4–15 | Drop | | 16–19 | ZRTP | | 20–63 | DTLS | | 64–79 | TURN channel if the source is a known TURN server, otherwise QUIC | | 80–127 | QUIC (short header, fixed bit set) | | 128–191 | RTP / RTCP | | 192–255 | QUIC (long header, fixed bit set) | Two consequences matter for a gateway: - Don’t classify with “long header OR fixed bit set.” That test, common in sketches, matches roughly three quarters of all possible first bytes. Any custom protocol whose first byte lands in 0x40–0xFF would be misrouted. RFC 9443 itself warns that a misimplemented classifier “could potentially misclassify packets.” - Endpoints behind a multiplexer must not send grease_quic_bit, because greasing breaks the first-byte scheme (RFC 9443, section 1). Designing the custom protocol so it can coexist If you control the game protocol, pick a first byte (a version or message-type field works) that stays in a range QUIC can’t occupy, such as 0–63, or 128–191 if you aren’t also carrying RTP. If you can’t change the protocol, you must accept that first-byte classification is a heuristic, add a stronger check on the long-header path (the Version field is 4 bytes at offset 1, and valid values include 0x00000001 for QUIC v1 and 0x6b3343cf for QUIC v2, RFC 9369), and treat anything ambiguous with care. A corrected decision procedure Datagram received │ ▼ byte0 in 64..127 or 192..255 ? ──no──► custom UDP pipeline │yes ▼ Long header (byte0 & 0x80)? ├─ yes ► check Version (v1, v2, or 0 for negotiation) ► QUIC └─ no ► short header: look up the DCID in your connection table ├─ hit ► same backend as the connection └─ miss ► drop, or send to QUIC backend (it may issue a stateless reset) The short-header branch is the catch. A passthrough gateway can only route short-header packets if it either remembers connection IDs (stateful) or the IDs themselves encode the routing decision (see section 5). Without one of those, it can only route by client address, which breaks when a client’s address changes. --- 4. Classifying TCP: ALPN, prefaces, and Encrypted Client Hello For TCP, the gateway peeks at the first bytes of the stream without consuming them (recv(..., MSG_PEEK) in C, bufio.Reader.Peek in Go), then forwards everything, including the peeked bytes, to the chosen backend. | First bytes | Meaning | Route | | --- | --- | --- | | 0x16 + TLS record header | TLS ClientHello | Parse ALPN (below) | | PRI (PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n) | Cleartext HTTP/2, “prior knowledge” (h2c), used by gRPC without TLS | HTTP/2 / gRPC backend | | GET , POST, HEAD, … | Cleartext HTTP/1.1 | HTTP backend | | anything else | Custom protocol | Fallback backend | ALPN routing. In TLS 1.3 the ClientHello is unencrypted, and it contains the ALPN extension (type 16) that lists the application protocols the client offers (h2, http/1.1, or custom tokens such as game-v1). The gateway reads the TLS record length from the 5-byte record header, peeks the whole record, walks the extensions, and routes on ALPN without terminating TLS. The Go implementation in section 7 does exactly this and is tested against real crypto/tls ClientHellos. The ECH caveat. RFC 9849 standardizes Encrypted Client Hello. With ECH, the client encrypts the sensitive ClientHello contents, which protects the SNI and can also protect ALPN, inside an inner ClientHello. The outer ClientHello carries innocuous values. A passthrough gateway therefore only sees what the client chose to expose in the outer hello. As ECH deployment grows, ALPN-based passthrough routing becomes less reliable, and the options are to terminate TLS at the gateway or to route on the outer name. Plan for this rather than assuming ALPN will always be visible. --- 5. Kernel-level demultiplexing: eBPF, XDP, and socket steering Peeking in user space costs a system call per decision and gives the kernel no hint about where to deliver. Two eBPF mechanisms address this, and they solve different problems. Option A: SK_REUSEPORT programs (route to the right socket) A BPF_PROG_TYPE_SK_REUSEPORT program replaces the default 4-tuple hash when picking a socket from a SO_REUSEPORT group. For UDP, the kernel patch that introduced it describes the use case directly: the program can peek into the datagram payload to learn application-level connection information, then choose the socket from a BPF map. This keeps packets in the normal kernel stack and lets you pin a QUIC connection to a specific worker by connection ID, even if the client’s address changes. Attach it with setsockopt using SO_ATTACH_REUSEPORT_EBPF (it requires CAP_SYS_ADMIN). Option B: XDP (act before the network stack) XDP runs at the driver level as soon as the NIC delivers a packet, before the kernel allocates its usual socket buffer. Meta’s open-source Katran layer-4 load balancer is built on this and describes running “right after packet has been received by network interface card … before kernel had any chance to run” in driver mode, with a generic (slower) mode as fallback. XDP can drop, pass, redirect to another interface, or redirect into an AF_XDP socket. AF_XDP matters for the design, because redirecting into it bypasses the kernel network stack: the packet lands in a user-space memory area, and you are responsible for everything the stack would have done (UDP checksums, ARP, ICMP, socket demultiplexing). The kernel documentation also specifies several behaviors that sketches often get wrong: - An XSKMAP is keyed by index, and a redirect only succeeds if the socket at that index is bound to the same device and receive queue the packet arrived on. Otherwise the packet is dropped. - If the map is empty at that index, the packet is dropped too. - The documented safe pattern is to look up the queue index first and XDP_PASS if no AF_XDP socket is bound. - If you want only certain traffic in user space, steer it to specific queues with NIC flow rules (the docs show ethtool -N … action <queue>). Here is a corrected XDP program that sends only QUIC-range datagrams on UDP port 443 to an AF_XDP socket and lets everything else (TCP, custom UDP, ARP, ICMP) continue through the kernel stack. It compiles with clang -target bpf; I have not load-tested it on a NIC. #include <linux/bpf.h> #include <linux/if_ether.h> #include <linux/in.h> #include <linux/ip.h> #include <linux/udp.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_endian.h> #define INGRESS_PORT 443 struct { __uint(type, BPF_MAP_TYPE_XSKMAP); __uint(max_entries, 64); __type(key, __u32); __type(value, __u32); } xsks_map SEC(".maps"); SEC("xdp") int xdp_quic_steer(struct xdp_md *ctx) { void *data_end = (void *)(long)ctx->data_end; void *data = (void *)(long)ctx->data; struct ethhdr *eth = data; if ((void *)(eth + 1) > data_end) return XDP_PASS; if (eth->h_proto != bpf_htons(ETH_P_IP)) return XDP_PASS; struct iphdr *iph = (void *)(eth + 1); if ((void *)(iph + 1) > data_end || iph->protocol != IPPROTO_UDP) return XDP_PASS; struct udphdr *udph = (void *)iph + iph->ihl * 4; if ((void *)(udph + 1) > data_end || udph->dest != bpf_htons(INGRESS_PORT)) return XDP_PASS; unsigned char *payload = (void *)(udph + 1); if ((void *)(payload + 1) > data_end) return XDP_PASS; /* RFC 9443 section 3: first byte 64..127 or 192..255 is QUIC * (64..79 is ambiguous only for traffic from a TURN server; not handled). */ unsigned char b0 = *payload; if (!((b0 >= 64 && b0 <= 127) || b0 >= 192)) return XDP_PASS; /* custom UDP: leave it to the kernel stack */ __u32 q = ctx->rx_queue_index; if (bpf_map_lookup_elem(&xsks_map, &q)) return bpf_redirect_map(&xsks_map, q, 0); return XDP_PASS; /* no AF_XDP socket on this queue */ } char _license[] SEC("license") = "GPL"; The program as written handles IPv4 without VLAN tags or IP fragmentation. A production version needs IPv6 and VLAN handling, and a decision about fragments. Routing QUIC by connection ID across hosts At scale, the question is how to route short-header packets without per-flow state. Two real approaches exist: - Server-ID routing. Meta described a stateless scheme at the Linux Plumbers Conference in 2021 where a server ID encoded into the identifiers lets the XDP load balancer route both TCP and QUIC without sharing state. - QUIC-LB. The QUIC working group’s draft “QUIC-LB: Generating Routable QUIC Connection IDs” defines a standardized way to encode routing information in server-chosen connection IDs. Note its status: as of this writing it is an expired Internet-Draft (latest revision 21, August 2025), not an RFC. It is a design reference, not a standard to claim compliance with. --- 6. Tunneling several protocols through one edge connection An edge tunnel (a persistent outbound connection from a private network to a public edge) can carry all three protocol classes over a single session, so the private side needs no inbound ports. PUBLIC EDGE (port 443) TCP streams ──┐ QUIC datagrams┼──► classifier ──► tunnel multiplexer ══ one outer session ══► UDP datagrams ┘ │ ▼ PRIVATE SIDE: demux by channel ► gRPC :50051 ► HTTP/3 :9001 ► game :7777 Use the right primitive for each traffic type - TCP and other byte streams map naturally onto streams (one multiplexed stream per accepted connection). Don’t describe them as “segments”; the tunnel carries a byte stream, not TCP segments. - UDP and QUIC are datagrams. Pushing them through a single reliable, ordered stream reintroduces head-of-line blocking and retransmission delay that real-time traffic is designed to avoid. Carry them in unreliable datagrams instead. Existing standards cover this: QUIC DATAGRAM frames (RFC 9221), HTTP Datagrams (RFC 9297), and CONNECT-UDP (RFC 9298, “Proxying UDP in HTTP”), which proxies UDP over HTTP/2 or HTTP/3 and can use QUIC datagrams when running over HTTP/3. If you can use one of these, you inherit interoperable framing and proxy support. - WireGuard is a layer-3 tunnel that carries IP packets over UDP, so it fits as a transport for raw packets, not as a “stream” substrate. An illustrative custom frame If you build your own framing, keep it small and explicit. This layout is an example design, not a standard: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Magic (0xA5) | Protocol Type | Channel ID | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Session Token | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Payload Length | Payload Data ... | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Field | Size | Purpose | | --- | --- | --- | | Magic | 8 bits | Frame-boundary sanity check (0xA5) | | Protocol Type | 8 bits | 0x01 stream data, 0x02 UDP datagram, 0x03 QUIC datagram | | Channel ID | 16 bits | Which backend service or tenant | | Session Token | 32 bits | Per-client identifier for state lookup | | Payload Length | 16 bits | Up to 65,535 bytes, enough for the largest UDP payload | A 32-bit token is a lookup key, not a credential. Anything that authenticates a tunnel client must come from a real handshake (TLS, a signed token, or WireGuard keys), never from a guessable counter. --- 7. Port exhaustion, session state, and what really limits you “Port exhaustion” gets blamed for many things. Here is what actually applies. The ephemeral port limit is per destination. On Linux the default outbound range is 32768–60999, about 28,232 ports. A connection is identified by its 4-tuple, so the same source IP and port can be reused for connections to different destinations. The practical limit appears when a gateway opens many connections from one source IP to a single backend IP and port: that tops out near the size of the range. Mitigations, roughly in order of preference: 1. Multiplex instead of one socket per client. Carry many client streams over a few long-lived backend connections (HTTP/2 or your own framing). Cloudflare’s write-up on ephemeral ports makes the same point: long-lived, multiplexed connections beat connection churn. 2. Spread across more 4-tuples. Use multiple backend addresses or ports, or multiple source IPs. 3. Widen or partition the range. Raise the range via net.ipv4.ip_local_port_range; newer kernels also offer a per-socket IP_LOCAL_PORT_RANGE option for carving up the range between components. Session tables. For connectionless traffic, the gateway needs a table that maps a client (or a connection ID) to its upstream path, with an idle timeout so entries expire. Keep it bounded and reap aggressively; an unbounded table is its own denial-of-service risk. TUN devices. Injecting raw IP packets through a TUN interface (the clone device is /dev/net/tun; the interface it creates is named something like tun0) forwards at layer 3, so your process allocates no per-client ephemeral ports. That is not the same as “bypassing OS port limits”: if the kernel then performs NAT or masquerading to reach the backend, the NAT still allocates source ports. Use TUN when you can route end to end without NAT. QUIC connection migration. QUIC separates a connection from the client’s IP and port using connection IDs (RFC 9000, section 5.1). Two details change how a gateway has to behave: - A migrating client switches to a new connection ID, not the same one on both paths. RFC 9000 requires connection IDs not to be linkable by observers and tells endpoints to limit each ID to one pair of addresses (sections 5.1 and 9.5). A gateway that tracks “this one CID maps to this backend” will therefore lose the connection when the client moves networks. - A passthrough gateway cannot read the new IDs, because NEW_CONNECTION_ID frames travel inside encrypted packets. The fix is for the server to issue routable IDs (server-ID routing or QUIC-LB, section 5), or for the gateway to terminate QUIC itself. RFC 9000 section 5.2.3 describes the fallback for simple address-based balancers: servers can send the disable_active_migration transport parameter, or use preferred_address to move clients to a dedicated address. Also note that zero-length connection IDs are fragile behind a shared address: RFC 9000 forbids using one IP and port for multiple concurrent zero-length-CID connections unless migration and NAT rebinding are ruled out. --- 8. A tested reference gateway in Go The following program implements the design above using only the standard library: - It listens on one port number for TCP and UDP. - For TCP it peeks the first bytes, parses the TLS ClientHello to read ALPN (h2 or http/1.1), recognizes the cleartext HTTP/2 preface and HTTP methods, and routes everything else to a fallback backend. Peeked bytes are replayed to the backend. - For UDP it classifies each datagram with the RFC 9443 ranges and keeps one upstream socket per client and backend pair, with idle expiry, relaying replies back through the listening socket. I compiled it, ran go vet, and tested it against local echo backends. The tests cover plaintext routing (PRI , GET , unknown bytes), UDP routing across the range boundaries (0xC0, 0x45 to QUIC; 0x05, 0x3F, 0x90 to the custom backend), and the ALPN parser against real crypto/tls ClientHellos (h2, http/1.1, an unknown protocol, and no ALPN). // Single-port TCP+UDP gateway: TCP routed by TLS ALPN / HTTP preface, // UDP routed by RFC 9443 first-byte ranges. Standard library only. package main import ( "bufio" "context" "io" "log" "net" "os" "os/signal" "sync" "syscall" "time" ) type Config struct { Listen string H2Backend string // gRPC / HTTP/2 HTTPBackend string // HTTP/1.1 TCPFallback string // anything else (custom TCP protocols) QUICBackend string // UDP, QUIC / HTTP/3 GameBackend string // UDP, custom protocol UDPIdle time.Duration } func main() { cfg := Config{ Listen: "0.0.0.0:8443", H2Backend: "127.0.0.1:50051", HTTPBackend: "127.0.0.1:8080", TCPFallback: "127.0.0.1:9000", QUICBackend: "127.0.0.1:9001", GameBackend: "127.0.0.1:7777", UDPIdle: 60 * time.Second, } ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() go serveTCP(ctx, cfg) go serveUDP(ctx, cfg) <-ctx.Done() } // ---------- TCP ---------- func serveTCP(ctx context.Context, cfg Config) { ln, err := net.Listen("tcp", cfg.Listen) if err != nil { log.Fatalf("tcp listen: %v", err) } go func() { <-ctx.Done(); ln.Close() }() for { c, err := ln.Accept() if err != nil { return } go handleTCP(c, cfg) } } func handleTCP(c net.Conn, cfg Config) { defer c.Close() br := bufio.NewReaderSize(c, 16*1024+5) c.SetReadDeadline(time.Now().Add(5 * time.Second)) // sniffing deadline backend := cfg.TCPFallback if hdr, err := br.Peek(5); err == nil { switch { case hdr[0] == 0x16: // TLS handshake record recLen := int(hdr[3])<<8 | int(hdr[4]) if rec, err := br.Peek(5 + recLen); err == nil { switch alpnPick(rec[5:]) { case "h2": backend = cfg.H2Backend case "http/1.1": backend = cfg.HTTPBackend } } case string(hdr[:4]) == "PRI ": // cleartext HTTP/2 prior knowledge backend = cfg.H2Backend case isHTTPVerb(hdr): backend = cfg.HTTPBackend } } c.SetReadDeadline(time.Time{}) up, err := net.DialTimeout("tcp", backend, 3*time.Second) if err != nil { log.Printf("dial %s: %v", backend, err) return } defer up.Close() var wg sync.WaitGroup wg.Add(2) go func() { defer wg.Done(); io.Copy(up, br); closeWrite(up) }() // br replays the sniffed bytes go func() { defer wg.Done(); io.Copy(c, up); closeWrite(c) }() wg.Wait() } func closeWrite(c net.Conn) { if cw, ok := c.(interface{ CloseWrite() error }); ok { cw.CloseWrite() } } func isHTTPVerb(b []byte) bool { for _, v := range []string{"GET ", "POST", "HEAD", "PUT ", "DELE", "OPTI", "PATC"} { if string(b[:4]) == v { return true } } return false } // alpnPick returns the first ALPN protocol the client prefers among h2/http/1.1, // or "" if the ClientHello has no ALPN extension (or cannot be parsed). func alpnPick(rec []byte) string { p := rec adv := func(n int) []byte { if len(p) < n { p = nil return nil } out := p[:n] p = p[n:] return out } if h := adv(4); h == nil || h[0] != 0x01 { // handshake type ClientHello + 3-byte length return "" } adv(2 + 32) // legacy_version + random if l := adv(1); l != nil { adv(int(l[0])) // session id } if l := adv(2); l != nil { adv(int(l[0])<<8 | int(l[1])) // cipher suites } if l := adv(1); l != nil { adv(int(l[0])) // compression methods } l := adv(2) if l == nil { return "" } exts := adv(int(l[0])<<8 | int(l[1])) for len(exts) >= 4 { typ := int(exts[0])<<8 | int(exts[1]) n := int(exts[2])<<8 | int(exts[3]) if len(exts) < 4+n { return "" } body := exts[4 : 4+n] exts = exts[4+n:] if typ != 16 || len(body) < 2 { // application_layer_protocol_negotiation continue } list := body[2:] for len(list) > 0 { k := int(list[0]) if len(list) < 1+k { return "" } switch s := string(list[1 : 1+k]); s { case "h2", "http/1.1": return s } list = list[1+k:] } } return "" } // ---------- UDP ---------- // classifyUDP follows RFC 9443 section 3: first byte 80..127 or 192..255 is QUIC; // 64..79 is QUIC unless it comes from a TURN server (not handled here). // Keep your own protocol's first byte in 0..63 or 128..191. func classifyUDP(first byte) (isQUIC bool) { return (first >= 64 && first <= 127) || first >= 192 } type udpSession struct { up *net.UDPConn seen time.Time } func serveUDP(ctx context.Context, cfg Config) { pc, err := net.ListenPacket("udp", cfg.Listen) if err != nil { log.Fatalf("udp listen: %v", err) } conn := pc.(*net.UDPConn) go func() { <-ctx.Done(); conn.Close() }() quicAddr, _ := net.ResolveUDPAddr("udp", cfg.QUICBackend) gameAddr, _ := net.ResolveUDPAddr("udp", cfg.GameBackend) var mu sync.Mutex sessions := map[string]*udpSession{} // key: client addr + backend go func() { // idle reaper t := time.NewTicker(cfg.UDPIdle / 4) defer t.Stop() for range t.C { mu.Lock() for k, s := range sessions { if time.Since(s.seen) > cfg.UDPIdle { s.up.Close() delete(sessions, k) } } mu.Unlock() } }() buf := make([]byte, 65535) for { n, client, err := conn.ReadFromUDP(buf) if err != nil { return } if n == 0 { continue } target := gameAddr if classifyUDP(buf[0]) { target = quicAddr } key := client.String() + ">" + target.String() mu.Lock() s := sessions[key] if s == nil { up, err := net.DialUDP("udp", nil, target) if err != nil { mu.Unlock() continue } s = &udpSession{up: up} sessions[key] = s go func(up *net.UDPConn, client *net.UDPAddr) { // backend -> client b := make([]byte, 65535) for { m, err := up.Read(b) if err != nil { return } conn.WriteToUDP(b[:m], client) } }(up, client) } s.seen = time.Now() mu.Unlock() s.up.Write(buf[:n]) } } What this gateway deliberately does not do It is a foundation, not a finished product. Before exposing it to the internet, plan for the following: - Client address preservation. Backends see the gateway’s address. Use the PROXY protocol for TCP, or carry the client address in your tunnel framing. - QUIC connection-ID awareness. Sessions are keyed by client address, so a client that migrates networks gets a new upstream socket. Combine it with server-issued routable connection IDs (section 5) if you need migration to survive. - One upstream socket per UDP client. That is simple and correct, but each socket uses an ephemeral port, so it inherits the limits in section 7. Cap session counts and consider the multiplexed designs there. - Abuse controls. There are no rate limits, connection limits, or amplification protection. QUIC servers must limit what they send to an unvalidated address to three times what they received (RFC 9000, section 8), and a gateway that relays replies to spoofed sources can be abused the same way. - Slowloris-style sniffing. The 5-second sniff deadline bounds idle connections, but you should also cap concurrent unclassified connections. - Fail-closed behavior. Unknown or malformed first bytes currently fall through to the fallback backend. Decide whether your deployment should drop them instead. --- 9. Measuring the benefit instead of assuming it Consolidating ports is mainly an operational win: fewer firewall rules, fewer listeners, one place to enforce policy. Performance gains are workload-specific, so measure before you claim any. Published or invented before-and-after numbers for “single-port versus multi-port” don’t transfer between environments, and the main mechanisms people cite need care: - Handshake latency. A single port does not shorten a TLS or QUIC handshake, and ALPN is carried inside the ClientHello, so it adds no round trip regardless of port count. QUIC 0-RTT applies only to resumed connections, and RFC 9000 warns it provides no protection against replay attacks, so it is not a general latency win and not safe for non-idempotent requests. - Per-packet cost. User-space classification costs a system call or two per datagram. Kernel-side classification (SK_REUSEPORT or XDP) removes that cost for the traffic it handles. Batched reads (recvmmsg, and UDP GRO and GSO on supporting kernels) cut it further without eBPF. XDP with AF_XDP trades kernel-stack features for speed, so it pays off at high packet rates, not at typical game-server loads. - What to measure. Track connection setup time at p50/p99, packets per second per core at a fixed loss target, CPU per Gbit/s, sockets and ephemeral ports in use (ss -s), and drops at each stage (NIC, socket buffers, your session table). Test with your real client mix, including QUIC clients that migrate and clients behind carrier NAT. --- 10. Implementation checklist - Sockets. Bind TCP and UDP to the same port number with ordinary listeners; add SO_REUSEPORT only for per-core scaling of one protocol. - UDP classification. Use the RFC 9443 first-byte ranges, never “long header or fixed bit.” Keep your own protocol’s first byte out of the QUIC ranges, and don’t enable grease_quic_bit on endpoints behind a first-byte multiplexer. - QUIC routing. Decide up front how short-header packets find their backend: server-issued routable connection IDs (server-ID routing, QUIC-LB), a connection table, or terminating QUIC at the gateway. Without one of these, migration breaks. - TCP classification. Peek, don’t consume; parse ALPN from the ClientHello; handle the PRI preface and HTTP methods; have a plan for ECH. - Kernel acceleration. Reach for SK_REUSEPORT eBPF first (it keeps the normal stack); use XDP and AF_XDP only when you need driver-level speed and can own the user-space stack. - Tunnels. Carry streams as streams and datagrams as datagrams (RFC 9221, RFC 9297, RFC 9298), and authenticate tunnel clients properly. - Port budgets. Count ephemeral ports per backend 4-tuple, not per host, and multiplex before you widen ranges. - Hardening. Bound every table, time out every session, rate-limit unvalidated UDP sources, and decide what happens to traffic you can’t classify. --- References - RFC 8999: Version-Independent Properties of QUIC - RFC 9000: QUIC: A UDP-Based Multiplexed and Secure Transport - RFC 9443: Multiplexing Scheme Updates for QUIC - RFC 9849: TLS Encrypted Client Hello - RFC 9298: Proxying UDP in HTTP - AF_XDP, The Linux Kernel documentation - BPF_PROG_TYPE_SK_REUSEPORT, eBPF documentation - QUIC-LB: Generating Routable QUIC Connection IDs (Internet-Draft) - Katran, Meta’s XDP-based layer 4 load balancer - From XDP to Socket, Linux Plumbers Conference 2021 - Cloudflare: How to stop running out of ephemeral ports - Peakhour: How browsers decide to use HTTP/3 --- Originally published at instatunnel.my

eBPF-Based Webhook Observability: Catching Dropped Packets Before the Application Layer Executive Summary: The Invisible Webhook Drop Problem Webhooks are the connective tissue of event-driven systems. Payment processors, code hosts and commerce platforms push state changes to subscriber backends over plain HTTP POST requests. Each provider has its own delivery rules, and those rules decide what a lost connection costs you: | Provider | Delivery behavior (per provider docs) | | --- | --- | | Stripe | In live mode, retries for up to three days with exponential backoff. In a sandbox, retries three times over a few hours. | | Shopify | Requires a 2xx response within five seconds. Retries failed deliveries up to eight times over about four hours, then removes the subscription if failures persist. | | GitHub | Records a failure if your server is down or takes longer than 10 seconds to respond. It does not automatically redeliver failed deliveries. | Retries can mask a kernel-level drop, but they also add delay and duplicate deliveries. With a sender that never retries, the event is simply lost. Application Performance Monitoring (APM) is the usual way to watch webhook endpoints, and it has a structural blind spot. SDK-based agents (Datadog, New Relic, OpenTelemetry SDKs) run inside your process. Even the newer eBPF-based auto-instrumentation, such as OpenTelemetry eBPF Instrumentation (OBI, donated by Grafana Labs as Beyla in 2025), observes requests that have already reached a process. A connection the kernel never hands to your application produces no span, no log line and no status code. [ Incoming Webhook Burst ] │ ▼ ┌───────────────────────────────────────────┐ │ Linux Kernel Network Stack │ │ ┌───────────────────────────────────────┐ │ │ │ NIC Rx ring → (XDP) → sk_buff alloc │ │ │ └──────────────────┬────────────────────┘ │ │ ┌──────────────────▼────────────────────┐ │ │ │ TCP: SYN queue → Accept queue │ │ ──► [ DROPPED HERE ] │ └──────────────────┬────────────────────┘ │ (APM never sees it) └────────────────────┼──────────────────────┘ │ accept() ▼ ┌───────────────────────────────────────────┐ │ User-Space Application Runtime │ │ ┌───────────────────────────────────────┐ │ │ │ Webhook worker / HTTP framework │ │ ──► Seen by APM (HTTP 2xx/5xx) │ └───────────────────────────────────────┘ │ └───────────────────────────────────────────┘ When the kernel discards a SYN or the final ACK of the handshake because a listening socket’s queues are full, the sender sees a connect timeout, a stalled request, or a reset. Your APM dashboard shows nothing unusual. This article shows how to find those drops with eBPF, how to measure the latency they hide, and where XDP does and does not help. It also covers the kernel and application tuning that prevents most of them. 1. Anatomy of a Kernel-Level Webhook Drop 1.1 The ingress path 1. NIC and DMA. Packets land in a receive ring buffer on the network interface card (NIC). 2. XDP (optional). If an XDP program is attached in native mode, it runs in the driver’s receive path, before the kernel allocates an sk_buff. 3. NAPI and SoftIRQ. The driver polls the ring and the kernel builds a struct sk_buff for each packet. 4. IP and TCP processing. The stack validates the packet and looks up the listening socket. 5. Listener queues. New connections pass through the SYN queue and then the accept queue. 6. accept(). Only now does your application see the connection, whether it is a Node.js event loop, a Go server or a Python worker. 1.2 The two queues where webhooks die As Cloudflare describes in SYN packet handling in the wild, every listening TCP socket has two queues: - The SYN queue holds half-open connections (state SYN_RECV) that have been sent a SYN+ACK and are waiting for the client’s final ACK. - The accept queue holds fully established connections waiting for the application to call accept(). The maximum length of both queues derives from the backlog argument the application passes to listen(2). The kernel silently caps that value at net.core.somaxconn. The default for somaxconn was 128 before Linux 5.4 and has been 4096 since 5.4. The default for tcp_max_syn_backlog scales with machine memory, and the 5.4 kernel raised its maximum default from 2048 to 4096. Accept queue overflow is the common webhook killer. If your runtime is busy, whether from a blocked event loop, an exhausted worker pool or a GC pause, it calls accept() late and the accept queue fills. Per Cloudflare’s analysis, once the accept queue is full, the kernel drops both inbound SYN packets and inbound final-handshake ACK packets. The counters that move are TcpExtListenOverflows and TcpExtListenDrops. What the sender experiences depends on net.ipv4.tcp_abort_on_overflow: - 0 (default). The kernel silently drops the final ACK. From the client’s side the handshake looks complete. Its request data goes unacknowledged until the server’s SYN+ACK retransmission timer and the client’s own retransmissions eventually line up with free space in the queue, or the connection times out. Dropped SYNs appear to the client as connect timeouts that are retried with exponential backoff. - 1. The kernel sends a RST after the final ACK, and the sender sees a connection reset. Cloudflare’s advice is that it is better to leave this setting alone, because resets make overload worse for clients that would otherwise retry. SYN queue overflow is rarer than it used to be. Linux enables SYN cookies by default (net.ipv4.tcp_syncookies=1). When a listener’s SYN queue is full, the kernel answers with a stateless SYN cookie instead of storing the request, so legitimate connections generally survive a SYN flood. Plain SYN queue drops still show up in TCPReqQFullDrop-style counters when cookies are disabled or unavailable. Either way, the application receives zero bytes for the affected connection, so an APM agent has nothing to record. 2. Tracing Ingress Drops with eBPF Counters like nstat -az TcpExtListenOverflows TcpExtListenDrops tell you that drops happened, but not which source, port or connection was hit. For a quick check on a live host, ss -ltn is the fastest tool. For listening sockets, Recv-Q is the current accept queue depth and Send-Q is its configured maximum. A Recv-Q pinned at Send-Q means the application is not accepting fast enough. eBPF adds per-event context without recompiling the kernel or attaching intrusive debuggers. 2.1 The skb:kfree_skb tracepoint and drop reasons Since Linux 5.17, kfree_skb carries a drop reason that explains why a socket buffer was discarded. These reasons are available in RHEL 8.8 and 9.2 and later, and the tracepoint is the main interface for reading them. Coverage has grown over time but is not complete. Kernel patches in 2022 added reasons for accept queue overflow and request queue full, originally named LISTENOVERFLOWS and TCP_REQQFULLDROP. Enum names have been reorganized in later releases, so check the symbols your kernel actually exposes in /sys/kernel/tracing/events/skb/kfree_skb/format. On older kernels, some listen-queue drops are not reported through kfree_skb at all, because the packet is freed as if it had been consumed normally. If a trace shows nothing while ListenDrops keeps climbing, this is the likely reason. Rely on the counters, and use the kprobe approach in Script 2. > A note on the snippets below. They are illustrative sketches. They assume a kernel with BTF (CONFIG_DEBUG_INFO_BTF), bpftrace 0.21 or later (older releases use args->field instead of args.field), and IPv4 traffic. Kprobe targets are not a stable ABI, so confirm the function names exist on your kernel (bpftrace -l 'kprobe:tcp_v4_*') and test in staging before running them in production. Script 1: Who is being dropped, and why This script counts dropped inbound TCP packets for webhook ports, by source address and drop reason. #!/usr/bin/env bpftrace tracepoint:skb:kfree_skb /args.protocol == 0x0800/ { $skb = (struct sk_buff *)args.skbaddr; $iph = (struct iphdr *)($skb->head + $skb->network_header); if ($iph->protocol == 6) { // TCP $tcp = (struct tcphdr *)($skb->head + $skb->transport_header); $dport = bswap($tcp->dest); if ($dport == 8080 || $dport == 443) { @drops[ntop($iph->saddr), $dport, args.reason] = count(); } } } interval:s:10 { time("%H:%M:%S drops by (source, dport, reason):\n"); print(@drops); clear(@drops); } The reason field prints as a number. Map it to a name using the format file mentioned above. Packets freed before the TCP header is parsed may carry an unset transport_header, so treat results as indicative rather than exhaustive. Script 2: Accept queue pressure per listening port The kernel considers the accept queue full when sk_ack_backlog exceeds sk_max_ack_backlog. The check runs on the listener socket in both tcp_v4_conn_request (incoming SYN) and tcp_v4_syn_recv_sock (final ACK). Both functions receive the listener as their first argument, so one probe covers both paths. #!/usr/bin/env bpftrace kprobe:tcp_v4_conn_request, kprobe:tcp_v4_syn_recv_sock { $sk = (struct sock *)arg0; @max_depth[$sk->sk_num] = max($sk->sk_ack_backlog); if ($sk->sk_ack_backlog > $sk->sk_max_ack_backlog) { @queue_full[probe, $sk->sk_num] = count(); } } interval:s:10 { time("%H:%M:%S\n"); print(@max_depth); print(@queue_full); clear(@max_depth); clear(@queue_full); } If you would rather not maintain probes, BCC and libbpf-tools ship maintained utilities for this area. tcpdrop, tcpretrans and tcpsynbl cover drops, retransmissions and SYN backlog depth. 3. Correlating Kernel Waits with Latency Spikes When the kernel drops a SYN, the sender waits for its retransmission timer. Per RFC 6298, the initial retransmission timeout is 1 second, and each failed attempt doubles it: RTO_n = RTO_0 × 2^n (RTO_0 = 1 s for the initial SYN) So a client’s SYN retries leave at roughly 1 s, 3 s, 7 s, 15 s and so on after the first attempt. Linux’s tcp_syn_retries setting caps how many times a client tries before giving up. Once a connection has RTT measurements, the minimum RTO in Linux is 200 ms, so established connections recover faster than new ones. A first-SYN drop is therefore especially expensive. A webhook whose first SYN is dropped and whose retry succeeds arrives about a second late, even though your handler runs in milliseconds: Sender (Webhook Provider) Receiver (Your Pod) ───────────┬─────────── ───────────────┬─────────────── │ 1. SYN │ │─────────────────────────────────────────────────►│ ──► Dropped (accept queue full) │ │ │ [ sender waits: initial RTO ≈ 1 s ] │ │ │ │ 2. SYN (retransmission #1) │ │─────────────────────────────────────────────────►│ ──► Accepted │ 3. SYN-ACK / 4. ACK │ │◄────────────────────────────────────────────────►│ │ 5. HTTP POST /webhook │ │─────────────────────────────────────────────────►│ ──► Handled in ~15 ms ▼ ▼ [ Sender-side latency ≈ 1,015 ms ] [ APM duration ≈ 15 ms ] This matters because the delay is short enough to hide inside a retry policy but long enough to cross a hard deadline. Shopify’s five-second response limit and GitHub’s ten-second limit are both reachable after two or three SYN retransmissions. Measuring the hidden wait You can measure this on the receiver. The idea is to remember when each client’s first SYN was dropped, then measure the gap when the same client port tries again. That gap is the retransmission wait your APM never sees. #!/usr/bin/env bpftrace tracepoint:skb:kfree_skb /args.protocol == 0x0800/ { $skb = (struct sk_buff *)args.skbaddr; $iph = (struct iphdr *)($skb->head + $skb->network_header); if ($iph->protocol == 6) { $tcp = (struct tcphdr *)($skb->head + $skb->transport_header); if (bswap($tcp->dest) == 8080 && $tcp->syn == 1 && $tcp->ack == 0) { if (@first_drop[$iph->saddr, $tcp->source] == 0) { @first_drop[$iph->saddr, $tcp->source] = nsecs; } @syn_drops = count(); } } } kprobe:tcp_v4_conn_request { $skb = (struct sk_buff *)arg1; $iph = (struct iphdr *)($skb->head + $skb->network_header); $tcp = (struct tcphdr *)($skb->head + $skb->transport_header); if (bswap($tcp->dest) == 8080) { $t0 = @first_drop[$iph->saddr, $tcp->source]; if ($t0 != 0) { @syn_retry_gap_ms = hist((nsecs - $t0) / 1000000); delete(@first_drop[$iph->saddr, $tcp->source]); } } } A histogram with clusters near 1,000 ms, 2,000 ms and 4,000 ms is the signature of SYN retransmission backoff. Bpftrace maps have a bounded size, so for long-running production use, port this logic to a libbpf program with an LRU hash map. Then export the histogram with a tool such as Cloudflare’s ebpf_exporter. Metrics worth alerting on You do not need a custom exporter to start. The Prometheus node_exporter netstat collector exposes the relevant kernel counters by default: - node_netstat_TcpExt_ListenOverflows and node_netstat_TcpExt_ListenDrops track accept queue overflows and listener drops. - node_netstat_TcpExt_TCPSynRetrans tracks SYN retransmissions. - node_netstat_TcpExt_SyncookiesSent shows when a SYN queue overflowed and cookies kicked in. Alert on any sustained increase in ListenDrops. Compare the rate of change to your APM request rate, and a divergence tells you traffic is arriving that your application never sees. 4. Mitigation Patterns: Where eBPF/XDP Helps and Where It Doesn’t Tracing finds the problem. Fixing it is mostly about how fast your application accepts connections, and secondarily about what the kernel does under load. 4.1 First, fix the queue 1. Raise the effective backlog. The kernel uses the smaller of the application’s backlog and net.core.somaxconn, so raise both. Frameworks differ: Node.js’s server.listen() defaults to a backlog of 511, and Go’s standard library derives its default from somaxconn. 2. Know the container caveat. net.core.somaxconn is namespaced per container. In Kubernetes it is classed as an unsafe sysctl, so setting it from a pod’s securityContext.sysctls requires the kubelet to allow it with --allowed-unsafe-sysctls. 3. Accept fast, process later. Shopify’s own guidance is to delay processing until after you have sent a response. A small receiver that verifies the signature, enqueues the payload to a durable queue (Kafka, SQS) and returns 2xx keeps accept() and response times short, even when downstream work is slow. 4.2 What XDP is good for XDP is a driver-level eBPF hook. In native mode it runs before sk_buff allocation, so dropping a packet costs very little. In Cloudflare’s benchmark, XDP_DROP discarded roughly 10 million packets per second on a single core. Newer drivers and hardware report substantially higher figures. Dropping at the socket or in the application costs far more per packet, and that gap is why XDP is used for DDoS mitigation at Cloudflare and for L4 load balancing in Meta’s Katran. That makes XDP a strong fit for removing traffic you never wanted before it can compete with real webhooks for CPU and queue slots. It is a poor fit for rate-limiting legitimate webhook senders. Do not rate-limit webhook providers per source IP at the packet level. Providers send from a small set of addresses. Stripe, for example, publishes a list of about a dozen webhook IPs, and GitHub publishes hook ranges through its /meta API. A per-IP, per-packet limiter would throttle your most important traffic, including its retransmissions and the data packets of a single request. A safer use is a blocklist, or the inverse, an allowlist. Drop traffic to your webhook port that does not come from provider-published ranges: #include "vmlinux.h" #include <bpf/bpf_helpers.h> #include <bpf/bpf_endian.h> #define ETH_P_IP 0x0800 struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 65536); __type(key, __u32); // IPv4 source address __type(value, __u8); } blocklist SEC(".maps"); SEC("xdp") int xdp_blocklist(struct xdp_md *ctx) { void *data = (void *)(long)ctx->data; void *data_end = (void *)(long)ctx->data_end; struct ethhdr *eth = data; if ((void *)(eth + 1) > data_end) return XDP_PASS; if (eth->h_proto != bpf_htons(ETH_P_IP)) return XDP_PASS; struct iphdr *ip = (void *)(eth + 1); if ((void *)(ip + 1) > data_end) return XDP_PASS; if (ip->protocol != 6) // TCP only return XDP_PASS; __u32 src = ip->saddr; if (bpf_map_lookup_elem(&blocklist, &src)) return XDP_DROP; return XDP_PASS; } char LICENSE[] SEC("license") = "GPL"; For provider ranges (CIDR blocks rather than single addresses), use a BPF_MAP_TYPE_LPM_TRIE instead of a hash map. A user-space controller can refresh the map from the providers’ published lists. Three limits to keep in mind: - XDP cannot read TLS-protected HTTP. It sees packet headers, not webhook payloads, endpoints or signatures. Anything that needs the HTTP layer still belongs in your ingress or application. - Native XDP needs driver support. Generic (SKB-mode) XDP works anywhere but runs after sk_buff allocation, losing most of the performance advantage. - XDP_REDIRECT does not forward to another node. BPF_MAP_TYPE_DEVMAP redirects to another network device on the same host, and BPF_MAP_TYPE_CPUMAP redirects to another CPU. Steering overflow traffic to a buffering tier on a different machine needs an encapsulating load balancer (Katran uses IPIP encapsulation) or a normal L4/L7 proxy in front of your workers. It is not something a stock XDP program does on its own. 5. Production Checklist for SREs | Layer | Recommended action | Why it matters | | --- | --- | --- | | Kernel tuning | Confirm net.core.somaxconn (4096 default on 5.4+, 128 before) and net.ipv4.tcp_max_syn_backlog. Keep tcp_syncookies=1 and leave tcp_abort_on_overflow=0 unless you have measured a reason to change it. | Prevents low queue limits on older kernels from dropping bursts. | | Application | Set an explicit listen backlog, and return 2xx quickly. Move processing off the request path. | Keeps accept() fast; this is the only fix for the root cause of accept queue overflow. | | Kubernetes | Allow and set net.core.somaxconn per pod where needed (unsafe sysctl). | The setting is per network namespace, so node-level tuning does not reach pods. | | Counters | Alert on node_netstat_TcpExt_ListenDrops, ListenOverflows and TCPSynRetrans. | Cheapest possible signal that the kernel is dropping what the app never sees. | | eBPF tracing | Use skb:kfree_skb (kernel 5.17+ for drop reasons) and listener-queue kprobes; try BCC/libbpf-tools tcpdrop, tcpretrans, tcpsynbl. | Adds source, port and reason context to the raw counters. | | Edge filtering | Use XDP blocklists or provider-range allowlists for webhook ports; do not packet-rate-limit providers. | Removes unwanted traffic cheaply without harming real senders. | | Resilience | Reconcile against provider APIs (GitHub’s delivery API, Stripe’s event list), and make handlers idempotent. | Covers deliveries lost without retry, and duplicates created by retry. | Conclusion Application-layer observability starts after accept(). The drops that matter most during a webhook burst happen before it, in listener queues the application cannot see. eBPF closes that gap: kernel counters tell you something is wrong, kfree_skb drop reasons and listener-queue probes tell you where and why, and a retransmission-gap histogram turns invisible kernel waits into a number you can alert on. The fix usually lives in the application and its queue sizing, with XDP as an efficient front-line filter for traffic you do not want. Add idempotent handlers and provider-side reconciliation, because providers differ sharply in whether they retry at all. Sources and Further Reading - Cloudflare, SYN packet handling in the wild - Cloudflare, How to drop 10 million packets per second - Red Hat, How to retrieve packet drop reasons in the Linux kernel - Linux man-pages, listen(2) - IETF, RFC 6298: Computing TCP’s Retransmission Timer - Stripe docs: webhooks and IP addresses - GitHub docs, Handling failed webhook deliveries - Shopify docs, Troubleshoot webhooks - Grafana Labs, OpenTelemetry and Grafana Labs: what’s new in 2026 --- Originally published at instawebhook.com