Hello eBPF: XDP-based Packet Filter (9)

Welcome back to my series on ebpf. In the last blog post, we learned how annotation processors can generate C code, simplifying writing eBPF applications.

This week, we'll use this work together with new support for XDP to create a simple package blocker for eBPF (GitHub):

./run_bpf.sh XDPPacketFilter twitter.com

This blocks all incoming IPv4 packages from twitter.com. We see how it works in this blog post. First, we start with some background on networking and explain what XDP is.

Network Packet

All networking is packet-based, with multiple layers of protocol from shared medium (e.g., Ethernet) to application level (e.g., HTTP):

Ethernet is the lowest-level protocol, with all packets coming to and from network interfaces being ethernet packets. The ethernet header contains the "physical" MAC address of both the source and destination of the package, combined with the protocol number of the next level protocol. We can represent it in C as follows:

struct ethhdr {
    unsigned char h_dest[6];
    unsigned char h_source[6];
    __be16 h_proto;
};

Today, Ethernet is routed on switch level, but it was initially used to communicate between devices that shared the same medium, typically cable.

Above the Ethernet protocol sit multiple protocols, but we're focusing here on the Internet Protocol (IP) with protocol type 0x0800. The IP protocol comes in two common variants, IPv4 and IPv6, and is used to communicate between devices on the whole internet. Although IPv6 has many advantages, IPv4 is still commonly used, and we're focusing on this variant in the following section to keep it simple. IP datagrams are typically fragmented into multiple smaller IP packets. An IPv4 header consists of the following parts:

struct iphdr {
    __u8 ihl: 4; // number of 32-bits in the header
    __u8 version: 4; // 4 (IPv4), 6 (IPv6)
    __u8 tos; // "priority" of the packet
    __be16 tot_len; // size of packet (header + data) in bytes
    __be16 id; // id of the datagram that this fragment belongs to
    // offset of this packet fragment in the unfragmented datagram
    __be16 frag_off;
    // hop count, router decrement it and drop package if ttl = 0
    __u8 ttl;
    // next level protocol
    __u8 protocol;
    // checksum, so that sum of all header 32-bit words is 0xFFFF
    __sum16 check;
    // source and destination address
    // (might be changed in transit 
    //  due to network address translation)
    struct {
        __be32 saddr;
        __be32 daddr;
    } addrs;
};

This misses the last field officially specified field, the options field, but it is, according to Wikipedia, usually not used:

The options field is not often used. Packets containing some options may be considered as dangerous by some routers and be blocked.

wikipedia

Above the IP protocol is the TCP protocol, which essentially adds ports and acknowledged package delivery, and on the web, the topmost layer is usually HTTP, which adds URL paths and more.

eXpress Data Path (XDP)

XDP is one of the most essential parts of the eBPF kernel land. It allows users to write firewalls, load balancers, and more, such as the packet filter of this blog post. To quote Jonathan Corbet:

The core idea behind the XDP initiative is to get the network stack out of the way as much as possible. While the network stack is highly flexible, XDP is built around a bare-bones packet transport that is as fast as it can be. When a decision needs to be made or a packet must be modified, XDP will provide a hook for a user-supplied BPF program to do the work. The result combines minimal overhead with a great deal of flexibility, at the cost of a little "some assembly required" label on the relevant man pages.

Accelerating networking with AF_XDP

The eBPF hooks attached to a specific network interface can inspect and modify the incoming packages, let them pass, drop, or send them back. A basic eBPF program that drops all packages looks, for example, like the following:

SEC("xdp")
int xdp_drop(struct xdp_md *ctx) {
    return XDP_DROP:
}

But please don't attach this program, as it would also drop Address Resolution Protocol (ARP) packages, which other members of your local ethernet network can map IP addresses to MAC addresses. Dropping all ARP packages can effectively disconnect your machine from the local network.

The passed xdp_md object contains the package content and some metadata:

struct xdp_md {
    __u32 data;
    __u32 data_end;
    __u32 data_meta;
    __u32 ingress_ifindex;
    __u32 rx_queue_index;
    __u32 egress_ifindex;
};

We're just focusing on the content that can be found between data and data_end. In fact, the header data structures I showed you in the previous section are precisely the structures that describe the content.

Armed with this knowledge, we can now create a package filter:

Writing a Packet Filter

The basic structure of our packet filter application consists of a Java part that handles the configuration and logging and an eBPF part that uses an XDP hook that is called for every received packet. As explained above, The XDP hook decides what to do with every packet. So the structures are as follows:

We start with the definition of eBPF for collecting statistics, blocked packets per IP address, and the configuration of the blocked IP addresses:

@BPFMapDefinition(maxEntries = 256 * 4096)
BPFHashMap<Integer, Boolean> blockedIPs;

@BPFMapDefinition(maxEntries = 256 * 4096)
BPFHashMap<Integer, Integer> blockingStats;

Now we move on to the eBPF program that checks for the IPv4 addresses and drops the packet if the address is in the blockedIPs map (based on the program from a blog post of sematext):

#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

// protocol numbers 
// copied from the linux kernel
#define ETH_P_8021Q 0x8100
#define ETH_P_8021AD 0x88A8
#define ETH_P_IP 0x08
#define ETH_P_IPV6 0x86DD
#define ETH_P_ARP 0x0806

SEC("xdp")
int xdp_pass(struct xdp_md *ctx) {
    // the package  
    void *end = (void *)(long)ctx->data_end;
    void *data = (void *)(long)ctx->data;
    u32 ip_src;
    u64 offset;
    u16 eth_type;

    struct ethhdr *eth = data;
    offset = sizeof(*eth);

    if (data + offset > end) {
        // ethernet package header is incomplete
        return XDP_ABORTED;
    }
    eth_type = eth->h_proto;

    /* handle VLAN tagged packet */
    // we use bpf_htons for the check to convert
    // from hardware to network endianess
    if (eth_type == bpf_htons(ETH_P_8021Q) || 
          eth_type == bpf_htons(ETH_P_8021AD)) {
        struct vlan_hdr *vlan_hdr;

        vlan_hdr = (void *)eth + offset;
        offset += sizeof(*vlan_hdr);
        if ((void *)eth + offset > end) {
            // ethernet package header is incomplete
            return false;
        }
        eth_type = vlan_hdr->h_vlan_encapsulated_proto;
    }

    /* let's only handle IPv4 addresses */
    if (eth_type != bpf_htons(ETH_P_IP)) {
        return XDP_PASS;
    }
    
    // get the IPv4 header
    struct iphdr *iph = data + offset;
    offset += sizeof(struct iphdr);
    
    // make sure the bytes you want to read are 
    // within the packet's range before reading them
    if (iph + 1 > end) {
        return XDP_ABORTED;
    }
    ip_src = iph->saddr;

    // find entry in block list
    void* ret = (void*)bpf_map_lookup_elem(&blockedIPs, &ip_src);
    if (!ret) {
        // IP not in blocked list
        return XDP_PASS;
    }

    // count the number of blocked packages per IP address
    s32* counter = bpf_map_lookup_elem(&blockingStats, &ip_src);
    if (counter) {
        // use atomics to prevent a race condition when a packet
        // from the same IP address is received on two
        // different cores at the same time
        // (thanks Dylan Reimerink for catching this bug)
        __sync_fetch_and_add(counter, 1);
    } else {
        u64 value = 1;
        bpf_map_update_elem(&blockingStats, &ip_src, 
                            &value, BPF_ANY);
    }

    return XDP_DROP;
}

Now we use it with some picocli-based command line handling to build our application:

@BPF(license = "GPL")
@Command(name = "XDPPacketFilter", 
         mixinStandardHelpOptions = true, 
         description = "Use XDP to block " +
                       "incoming IPv4 packages from a URLs")
public abstract class XDPPacketFilter 
  extends BPFProgram implements Runnable {

    // maps, ...

    private static final String EBPF_PROGRAM = """
            // ...
            """;

    @Parameters(arity = "1..*", description = "URLs to block")
    private String[] blockedUrls;

    @Option(names = "--run-url-retrieve-loop", 
        description = "Try to retrieve the content " + 
                      "of the first URL in a loop")
    private boolean runURLRetrieveLoop;
   
    private Map<Integer, String> ipToUrlMap;
    
    void setupBlockedIPMap() {
        ipToUrlMap = Arrays.stream(blockedUrls).flatMap(url -> {
            try {
                // Resolve the URL to the related IP addresses
                return Arrays.stream(
                    InetAddress.getAllByName(url))
                               .map(addr -> 
                       // convert the IP address to numbers
                       Map.entry(XDPUtil.ipAddressToInt(addr), url));
            } catch (UnknownHostException e) {
                throw new RuntimeException(e);
            }
        }).collect(Collectors.toMap(
             Map.Entry::getKey, 
             Map.Entry::getValue));
        ipToUrlMap.keySet().forEach(ip -> {
            // put the IP addresses in the map
            blockedIPs.put(ip, true);
        });
    }
    
    // print the content of blockingStats
    void printBlockedLog() {
        out.println("Blocked packages:");
        blockingStats.forEach((ip, count) -> {
            out.println("  Blocked " + count + " packages from " +
                    XDPUtil.intToIpAddress(ip) +
                    " (" + ipToUrlMap.get(ip) + ")");
        });
    }

    @Override
    public void run() {
        setupBlockedIPMap();
        if (runURLRetrieveLoop) {
            XDPUtil.openURLInLoop(blockedUrls[0]);
        }
        xdpAttach(getProgramByName("xdp_pass"), 
            XDPUtil.getNetworkInterfaceIndex());
        // print the blocking statistics every second
        while (true) {
            printBlockedLog();
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                throw new RuntimeException(e);
            }
        }
    }

    public static void main(String[] args) {
        try (XDPPacketFilter program = 
              BPFProgram.load(XDPPacketFilter.class)) {
            var cmd = new CommandLine(program);
            cmd.parseArgs(args);
            if (cmd.isUsageHelpRequested()) {
                cmd.usage(out);
                return;
            }
            program.run();
        }
    }
}

This is all we need, now we can use it:

# block twitter.com and log in the background
> ./run_bpf.sh XDPPacketFilter twitter.com > log.txt &
# try to access twitter.com with a timeout of 5 seconds
> wget twitter.com --timeout 5
URL transformed to HTTPS due to an HSTS policy
--2024-04-22 13:28:29--  https://twitter.com/
Resolving twitter.com (twitter.com)... 104.244.42.65
Connecting to twitter.com (twitter.com)|104.244.42.65|:443... failed: Connection timed out.
Retrying.

--2024-04-22 13:28:36--  (try: 2)  https://twitter.com/
Connecting to twitter.com (twitter.com)|104.244.42.65|:443... failed: Connection timed out.
Retrying.

--2024-04-22 13:28:43--  (try: 3)  https://twitter.com/
Connecting to twitter.com (twitter.com)|104.244.42.65|:443... failed: Connection timed out.
Retrying.

# and so on

So we can't access twitter.com anymore till we stop our application.

Conclusion

Using XDP and eBPF, we can create a partial packet filter that is easily extended into a firewall and blocks incoming packets. The filtering overhead is low, as the packets are processed directly in the kernel. Using hello-ebpf, we can wrap the filter program in a neat command-line application.

But this is only the starting point; we can use a few more XDP features to create a fast load-balancer and add support for cgroups to create a proper firewall, filtering out-going packages too. The following two blog posts will be about this, so see you in two weeks' time.

This article is part of my work in the SapMachine team at SAP, making profiling and debugging easier for everyone. This article first appeared on my personal blog mostlynerdless.de.