<-- home

SPA with Netfilter Kernel Modules

I recently setup a DigitalOcean instance to host a project I've been working on, and since I'm exposing potentially sensitive data (API keys) on "the cloud" I wanted to ensure I locked it down as much as possible. Of course this meant all of the usual host provisioning such as enabling logging, SELinux configuration, firewall configuration, attack surface reduction (removing all unnecessary services), and of course automatic security updates. But as anybody who has put anything on the internet for more than 5 minutes knows, those logs will fill up quickly and I didn't want this particular server taking up too much of my time on a regular basis.

With only SSH exposed, many common solutions to reduce the amount of malicious traffic are to install rate limiting software like Fail2Ban and move to a non-standard port. And this actually works fairly well. But I've been curious about Port Knocking-like solutions for a while and wanted to dig a little deeper.

Particularly, I had recently learned a decent amount about Berkeley Packet Filters and Netfilter and felt it was a good opportunity to try and roll my own solution incorporating both. One that, particularly, wouldn't be susceptible to replay and MITM attacks like traditional port knocking solutions.

During the development I tried to make performance a priority, since a slow filtering process would severely impact the host if every incoming packet needed to go through every verification check. Luckily, BPF is kernel supported and extremely efficient at filtering. The idea is to use a unique enough packet as the "knock" so that 99% of the noise is filtered out by a BPF filter before it reaches any signature verification code. This article is an overview of my attempt.

All source code is located on github: DrawBridge.

Overview: Design

The DrawBridge kernel module is comprised of two components, a Netfilter NF_INET_LOCAL_IN hook and a separate kernel thread that uses Berkeley Packet Filters and raw sockets to bypass the TCP/IP stack and receive the "knock".

The raw socket thread is responsible for authenticating clients and adding their connection to a shared doubly linked list. The netfilter hook will then return either NF_ACCEPT or NF_DROP for new TCP session depending on whether the client has authenticated or not. The great part about this is that:

  • The hook will drop non-authenticated packets early in the Netfilter routing process.

  • The Netfilter hook hands the connection off to the rest of the Netfilter stack if it returns NF_ACCEPT, so the accepted connection will still be compliant with whatever firewall rules you have setup (i.e the connection to the SSH daemon can still be dropped later in the Netfilter stack if there is an iptables rule that drops the connection).


Netfilter is the main packet filtering subsystem within the Linux kernel's protocol stack. You're probably familiar with IPTables, which is built upon the Netfilter framework, so you have an idea of how Netfilter routing works. Logically similar to IPTables chains, Netfilter core provides five hooks that higher level modules can interface with. These hooks outline the flow of a frame/packet through the Netfilter protocol stack.

When a packet first arrives at an interface it is first received by the link-level drivers, which handle things like data-link layer cyclic redundancy checks (CRC) on Ethernet frames. Afterwards the kernel delivers the frame to the protocol stack or any raw sockets listening for packets. At this point NETFILTER receives the frame and the first hook is invoked: NF_IP_PRE_ROUTING.

If the packet is not dropped by the PRE_ROUTING hooks then routing decisions are made. This is where it is determined if the packet is destined for the local machine or not. If so it will be sent through NF_IP_LOCAL_IN, and if not it will be dropped unless the machine is configured for forwarding then it will be directed through the NF_IP_FORWARD and POST_ROUTING hooks.

For our port knocking purposes, we only really care about the packet if it's destined for the local machine on one of the ports we're monitoring, for that reason I chose to hook NF_IP_LOCAL_IN and not NF_IP_PRE_ROUTING.

Registering the Netfilter Hook

To actually register the first component of DrawBridge, the Netfilter hook, we first need to define an nf_hook_ops structure. Aptly named, this structure contains the options for our Netfilter hook. The nf_hook_ops struct is defined in linux/netfilter.h:

struct nf_hook_ops {
    struct list_head  list;

    /* User fills in from here down. */
    nf_hookfn   *hook;
    struct net_device *dev;
    void      *priv;
    u_int8_t    pf;
    unsigned int    hooknum;
    /* Hooks are ordered in ascending priority. */
    int     priority;

The struct members we're concerned with are:

  • nf_hookfn * hook: Pointer to our hook's callback function (Netfilter will call this function)

  • unsigned int hooknum: This is the hook table we're targeting (in our case NF_INET_LOCAL_IN)

  • u_int8_t pf: Protocol we want to hook, we'll support both IPv4 and IPv6

  • int priority: Since we think we're important enough, we'll give ourselves a high priority.

This structure can be defined both statically or dynamically. Since I'm only defining it this one time I chose to do it statically. Here is the IPv4 definition:

static struct nf_hook_ops pkt_hook_ops __read_mostly = {
  .pf     = NFPROTO_IPV4,
  .priority = 1,
  .hooknum  = NF_INET_LOCAL_IN,
  .hook   = &(pkt_hook),

pkt_hook in this case is the callback function we'll implement to drop NEW connections if the source computer hasn't "knocked". Registering the hook is actually incredibly simple, all it takes is a call to nf_register_hook:

ret = nf_register_hook(&(pkt_hook_ops));

if(ret) {
    printk(KERN_INFO "[-] Failed to register hook\n");
    return ret;

We just need to remember to unregister the hook when we unload the module. This can be done with the corresponding call:


Berkeley Packet Filters

The second component of the DrawBridge module listens on a raw socket and uses Berkeley Packet Filters to filter out extraneous packets/frames. Raw sockets as well as the Netfilter protocol stack will each receive a copy of a frame after it's received by the physical interface, regardless of the packet's protocol. Because of this, the raw socket will receive a packet even if NetFilter hooks/IPTables rules are explicitly configured to drop it. Which is pretty sweet, because then we don't need to create separate firewall rules for our "knock" packets to be received.

The original BPF paper was published in 1992 by Steven McCanne and Van Jacobson of the Lawrence Berkeley Laboratory. Their motivation in creating the BSD Packet Filter was to minimize crossing of the user-kernel protection boundary when using user-level packet capture applications. By implementing a kernel agent to efficiently filter packets on behalf of the user application, they are able to discard unwanted packets as early as possible and only copy the already filtered packets across the user-kernel boundary.

You might say; "Well okay Bradley, what's so cool about that?". Well I'll tell you. They didn't just port some everyday packet filtering code into the kernel and call it a day. They designed an entire virtual machine in the kernel that interprets BPF bytecode on the fly. And it's very efficient; Cloudfare has a few blog posts describing how they use BPF to quickly drop massive floods of packets at scale.

Thanks to the work they did on this, all we need to do is compile narrowly-enough defined BPF instructions, and we'll be able to quickly filter out all the noise on our raw socket. Tcpdump provides a very convenient way to do this:

// Compiled w/ tcpdump 'icmp[icmptype] == 8' -dd
struct sock_filter code[] = {
  { 0x28, 0, 0, 0x0000000c },
  { 0x15, 0, 8, 0x00000800 },
  { 0x30, 0, 0, 0x00000017 },
  { 0x15, 0, 6, 0x00000001 },
  { 0x28, 0, 0, 0x00000014 },
  { 0x45, 4, 0, 0x00001fff },
  { 0xb1, 0, 0, 0x0000000e },
  { 0x50, 0, 0, 0x0000000e },
  { 0x15, 0, 1, 0x00000008 },
  { 0x6, 0, 0, 0x00040000 },
  { 0x6, 0, 0, 0x00000000 },

Then it's just a matter of applying the filter to our raw socket. This socket interface is a little different in the kernel but it's similar enough. sock_create() is similar to socket() in user-mode, and sock_setsockopt() is the kernel equivalent of setsockopt(). SOCK_RAW will specify that we want a raw socket, and we'll use the SO_ATTACH_FILTER option to apply our BPF filter.

struct sock_fprog bpf = {
  .len = ARRAY_SIZE(code),
  .filter = code,

error = sock_create(PF_PACKET, SOCK_RAW, htons(ETH_P_ALL), &sock);

if (error < 0) {
  printk(KERN_INFO "[-] Could not initialize raw socket\n");
  return -1;

ret = sock_setsockopt(sock, SOL_SOCKET, SO_ATTACH_FILTER, 
                     (void *)&bpf, sizeof(bpf));

if(ret < 0) {
  printk(KERN_INFO "[-] Could not attach bpf filter to socket\n");
  return -1;

From the linux socket man page: "When protocol is set to htons(ETH_P_ALL), then all protocols are received. All incoming packets of that protocol type will be passed to the packet socket before they are passed to the protocols implemented in the kernel."

Authentication & Replay Defense

When the raw socket thread receives a packet that makes it through the BPF filter it will first parse the packet to ensure it conforms to the DrawBridge protocol specification and then verify the RSA signature of the request. A proper DrawBridge packet will have the following format:

Once received the signature and digest will be parsed from the packet, the DrawBridge protocol fields will be hashed and compared to the received digest, and then the signature will be verified:

// Parse the packet for a signature and digest
sig = get_signature(pkt);

if(!sig) {
    printk(KERN_INFO "[-] Signature not found in packet\n");

// Hash the TCP header + timestamp + port to unlock
hash = gen_digest((unsigned char *)&(res->tcp_h), 
                sizeof(struct packet) - (sizeof(struct ethhdr)
                + sizeof(struct iphdr)));

// Check that the hash matches the received digest
if(memcmp(sig->digest, hash, sig->digest_size) != 0) {
    printk(KERN_INFO "-----> Hash not the same\n");

// Verify the signature
if(!verify_sig_rsa(req, sig)) {

Knock State & Connection Tracking

After it's confirmed that a knock packet has authenticated properly, it will track the connection's source IP address in a custom state struct by adding it to a doubly linked list of non-expired connections:

typedef struct conntrack_state {

  // IP version type
  int type;

  // Destination port
  __be16 port;

  // Source IP
  union {
    __be32 addr_4;
    struct in6_addr addr_6;
  } src;

  // Timestamp
  unsigned long time_added;

  // List entry
  struct list_head list;

} conntrack_state;

When the Netfilter hook is called, it will check if the source address and destination port of the NEW connection has an existing knock state in the doubly linked list state->list. If this connection has been preceded by a properly authenticated knock state_lookup will return 1:

int state_lookup(conntrack_state * head, int type, __be32 src, 
                        struct in6_addr * src_6, __be16 port) {

  conntrack_state  * state;

  // Obtain a read lock 

  // Loop through the list
  list_for_each_entry_rcu(state, &(head->list), list) {

      // IPv4 check
      if(state->type == 4 && state->src.addr_4 == src 
                          && state->port == port) {

          return 1;

      // IPv6 check
      } else if (state->type == 6 
                      && ipv6_addr_cmp(&(state->src.addr_6), src_6) == 0 
                      && state->port == port) {

          return 1;

  // Release the lock

  return 0;

The Netfilter callback will return NF_ACCEPT if state_lookup evaluates to true, and NF_DROP otherwise for new connections that are destined to a port the user has configured knocking for.

if(state_lookup(knock_state,4, ip_header->saddr, NULL, tcp_header->dest))
    printk(KERN_INFO  "[!] Connection accepted      source:%s\n", src);
    return NF_ACCEPT;

return NF_DROP;


Just learned about fwknop from a friend. I haven't done an extensive comparison between the two, but their project definitely has a lot more development time and testing behind it. So I'd consider it a more production ready solution. One key difference between the two ideas is that their implementation doesn't hook into the protocol stack at a low enough level to drop packets before they hit the host firewall, and it can't receive an auth packet on any port because they don't use raw sockets. But they handle NAT better and have clients developed for a lot of systems. It's a cool project that's worth a look.


NetFilter Team: Writing Netfilter Modules

The BSD Packet Filter - Berkeley

Linux Kernel: public_key_verify_signature

Linux Kernel Mailing List Re: Typos and RSA