Learning eBPF the Hard Way: Starting from nginx eBPF
This post explains why I want to start this new series and why I like eBPF, then uses Reuseport eBPF as an entry point to analyze how it solves UDP hot upgrade problems. It also looks into nginx's eBPF implementation and its limitations, revisits the pitfalls I hit when I did not understand how eBPF loaders work, and finally digs into Cloudflare's udpgrm project.
Preface
A Serious Disclaimer About This New Series
First, the title sounds a little too ambitious, as if I have a deep understanding of eBPF. If you clicked in expecting advanced eBPF content, I should apologize first. I am still in the early stage of learning eBPF, and there is a long way to go. I only want to use this new series to record how I stumbled through learning eBPF, and to push myself to learn faster through practice.
In theory, eBPF should not be that hard to learn. I have also been reflecting on why I hit so many pitfalls. So this series is partly self-reflection and partly learning notes.
Why I Like eBPF
A Brief Note on eBPF
Given the title, this post will not spend any words introducing basic eBPF concepts. Otherwise it would not be "the hard way". You can ask AI for the basic concepts, and it will produce a long explanation.
When I first started learning eBPF, I directly found an ebook: learning-ebpf. It is a good book and suitable for getting started. I read it roughly at first. I had originally planned to read it carefully, but I kept running into unfamiliar context, so I decided to practice first and revisit the book as needed.
After learning the basics of eBPF, if you are still confused about Linux tracing systems, for example what mechanisms eBPF relies on, how it obtains the data you want, or when it takes effect, I recommend this post: Linux tracing systems & how they fit together. It gives a very intuitive high-level view. Even after using bcc or bpftrace, beginners can still have many questions because these convenient frameworks and tools hide too many low-level details.
What Kind of Bug Is Hardest to Debug?
Before talking about why I like eBPF, I want to revisit a familiar question: what kind of bug is hardest to debug?
Intuitively, people may think bugs whose root cause is not obvious, or bugs in unfamiliar domains, are hard. But from my own experience, I think bugs that are hard to reproduce are the hardest.
Debugging is mostly about getting more clues. The more clues you have, the easier root-cause analysis becomes. A bug that can be reproduced reliably gives us a steady stream of clues. We can use them to correct our direction and narrow the scope. The classic method is adding lots of logs along the reproduction path and studying the behavior step by step. Even if the problem is inside an unfamiliar third-party dependency, we can still gain confidence and locate it, by asking for help if possible or reading the code ourselves if not. At worst, we can narrow the problem to a small area.
But if a bug happens only in production with very low probability, and the information available at that moment is not enough to identify the root cause, debugging becomes especially painful. This is easy to understand. Most ways of obtaining more information are expensive. Temporarily modifying the version to add logs is already painful because the release process itself costs time. The logs are also hard to choose. Too many logs hurt production performance and cost. Too few logs may miss the key signal, wasting a full release cycle while the bug remains unresolved. New logs can also introduce performance and stability risks. Thinking about that, the fragile trust between you and your manager may crack a little more.
eBPF Helps a Lot Here
eBPF is designed around flexibility, programmability, low overhead, and safety. It can quickly and continuously collect information for hard-to-reproduce bugs, so it is very useful for this kind of problem. Once enough information is collected, there is usually no bug that remains impossible to solve. Many interesting debugging stories are really about debugging strategy and how to efficiently obtain the right information.
Of course, eBPF shines in many domains, and I will not list them all here. But my favorite thing about eBPF is that it can help debug difficult problems. I want to recommend an excellent technical blog series, A Tour of Dynamic Tracing. It explains this area very well. I reread it almost every year, and it still feels fresh. My interest in this area is not only because debugging is a core engineering ability, but also because of my previous work experience. You can read my earlier post, Incident Response: Lessons from 4+ Years On Call, to get a sense of that journey.
Unfortunately, in my first job I had almost no chance to use eBPF. The production kernel was too old, stuck at 3.10, and did not support eBPF. We could only use SystemTap in emergencies. But SystemTap is clearly not as nice as eBPF. I still remember one old incident where a temporary SystemTap script I wrote directly crashed a production machine. Another time it drove CPU usage to 100%. If I had been caught, that would have been a serious production-safety issue. Unless absolutely necessary, I will not casually use SystemTap again.
Reuseport eBPF
As mentioned above, when I first started learning eBPF, I felt the learning was too dry. I also knew that eBPF was one solution for correctly matching UDP-based transport protocol packets to specific processes. So I decided to learn eBPF through the question of how eBPF solves this problem.
Before reading the following sections, I recommend first reading my previous two posts, Implementing QUIC from Scratch with Rust: Connection Migration and Linux Server-Side UDP Network Programming: Zero-Loss Hot Upgrade, to understand the background.
The Linux Kernel Solution
In Linux Server-Side UDP Network Programming: Zero-Loss Hot Upgrade, I pasted a lot of Linux UDP sk matching and lookup code. If you read that post, you may have noticed that some eBPF-related logic appears briefly in the core path.
More specifically, in the Linux kernel source, eBPF inserts a hook into the core Reuseport matching logic. The program type is BPF_PROG_TYPE_SK_REUSEPORT. This means that when a UDP packet is matched in the kernel UDP two-tuple hash table, and a suitable Reuseport Group is found, if an eBPF program is attached to this hook, the matching logic defined in the eBPF program is executed. If the eBPF code gives up selecting a sk, the original logic is used: select a sk from the Reuseport Group according to the UDP four-tuple hash.
Next, let's see what the kernel gives BPF_PROG_TYPE_SK_REUSEPORT programs, so they can route UDP-based protocol packets to the correct sk.
In Implementing QUIC from Scratch with Rust: Connection Migration, I mentioned that UDP-based protocols usually do not use the UDP four-tuple as their connection identifier. This decouples them from UDP and lower layers. QUIC, for example, uses connection id as its connection identifier.
For QUIC, the eBPF program must parse the UDP payload and extract the protocol-defined QUIC connection id. Then, based on that connection ID, all UDP packets in the lifetime of a QUIC connection should be locked to one unique sk, even if the UDP four-tuple changes.
The BPF_PROG_TYPE_SK_REUSEPORT documentation says the eBPF program context is struct sk_reuseport_md, from which we can access key information such as UDP payload data. The program type also provides the BPF_MAP_TYPE_REUSEPORT_SOCKARRAY map type and the bpf_sk_select_reuseport helper.
The usage is straightforward. The map acts as a hash table. Its key is a 4-byte index, and its value is an sk with reuseport enabled, which we can think of as a socket. After creating the eBPF map, we manually insert the UDP listening sockets into it. Then, inside the eBPF program, we extract the key value from the UDP datagram and call bpf_sk_select_reuseport to choose the correct sk, meaning the unique UDP socket.
One practical detail: normally, implementations do not maintain a mapping from every QUIC connection to a UDP socket. That would require a large hash table for thousands or millions of connections. eBPF map sizes are fixed and do not support rehash expansion, likely for safety reasons. So the design is similar to the QUIC-LB idea mentioned in the connection migration post: each UDP Reuseport socket is inserted into the eBPF map once, and its key is fixed. Then the QUIC connection id only needs to carry that key according to a private convention. A 4-byte key normally does not consume the whole connection id, so it does not affect connection ID uniqueness. If this still feels vague, the nginx source analysis below will make it clearer.
Correct Usage from the Application Layer
Because we can write our own eBPF program to decide which Reuseport socket a UDP packet belongs to, we no longer need the Established-over-unconnected technique described in the UDP network programming post. We do not need to simulate TCP network programming by creating one independent UDP socket for every UDP-based transport connection and then hope that the kernel's UDP packet lookup logic always delivers packets for a QUIC connection to the same UDP socket.
As shown below, each process only needs to hold one UDP socket. For new QUIC requests, the eBPF program bypasses them and lets the kernel's default UDP four-tuple hash logic load-balance them across all listening processes, as shown in step 2. During the QUIC handshake, the connection updates its custom connection id so that the connection id contains the key corresponding to the eBPF map. Later UDP packets follow the eBPF program logic, extract the eBPF map key, and find their owning UDP socket.
Someone may ask: without eBPF or Established-over-unconnected, can QUIC still work well? After all, the kernel always uses the UDP four-tuple hash to select a sk in the Reuseport Group. But as discussed before, QUIC supports connection migration, and the UDP four-tuple is not the identifier of a QUIC connection. Once the UDP four-tuple changes, the default kernel distribution logic can break. The Reuseport Group is also not immutable. Hot upgrade changes the group, which changes the hash result. QUIC cannot accept that. Next, we analyze how nginx uses eBPF to solve this problem. Note that the diagram above is not complete enough. It can solve QUIC connection migration, but not hot upgrade.
nginx eBPF Implementation Details
After discussing the Reuseport eBPF principle, you probably still only have a rough idea of where Reuseport eBPF takes effect in the kernel and what it does. At least that was how I felt at the time. So the next step is to see how Reuseport eBPF is used in practice. I happened to see nginx introduce eBPF for the same problem, so I happily jumped into nginx's eBPF implementation details.
Questions While Reading the Source
When reading code, I like to first try to find problems, even in a well-known open-source project like nginx. Perhaps nitpicking is much easier than creating.
Where Is libbpf?
After a quick scan, nginx eBPF code appears mainly in three places:
src/core/ngx_bpf.cwraps thebpfsystem call and supports loading eBPF programs and CRUD operations on eBPF maps.src/event/quic/bpf/contains the eBPF program code to be loaded, written in C.src/event/quic/ngx_event_quic_bpf.cmaintains sockets for nginx Reuseport QUIC listening configuration and inserts them into the eBPF map.
My first question was: where is libbpf? Where are the bpf skeletons I had just learned? I had followed tutorials and written a few examples, but nginx showed none of the familiar traces.
I read the code more carefully and found the nginx eBPF build script. After reading it, I roughly understood nginx's approach. It compiles the eBPF program in src/event/quic/bpf/ngx_quic_reuseport_helper.c into eBPF bytecode with clang, extracts the eBPF bytecode from the relocatable object file, and generates src/event/quic/ngx_event_quic_bpf_code.c.
That feels a little like metaprogramming. No wonder libbpf is not needed. Pretty cool. I immediately thought I understood nginx's intention: reduce third-party dependencies and keep things clean.
Then I noticed the build script is not called by nginx's normal build process. So I concluded that src/event/quic/bpf/ only exists to help people understand the eBPF program logic. The file that really participates in eBPF functionality is src/event/quic/ngx_event_quic_bpf_code.c.
Possible Resource Leak?
Next, I habitually checked the shutdown flow. During hot upgrade, when old processes exit, how is the eBPF map handled? I was surprised to find that ngx_bpf_map_delete is never called in nginx. I suddenly wondered whether this leaks resources.
I checked again. During each hot upgrade, QUIC Reuseport sockets are created again, and new sockets are inserted into the eBPF map, while old sockets are not deleted from it. Then I thought: nginx cannot make such an obvious mistake. eBPF map size is fixed, so this would be easy to discover. Maybe this is the old trick where sockets are automatically removed from the eBPF map when they are closed.
I saw that nginx uses BPF_MAP_TYPE_SOCKHASH, which surprised me. I had assumed it would use BPF_MAP_TYPE_REUSEPORT_SOCKARRAY. The BPF_PROG_TYPE_SK_REUSEPORT documentation clearly says that since kernel 5.8, BPF_MAP_TYPE_SOCKHASH and BPF_MAP_TYPE_SOCKMAP can also be used with this helper.
Since v5.8
BPF_MAP_TYPE_SOCKHASHandBPF_MAP_TYPE_SOCKMAPmaps can also be used with this helper.
First, BPF_MAP_TYPE_SOCKHASH supports automatically cleaning map entries when sockets are finally closed. So there is no resource leak. To be safe, I started an nginx group and tested it. The method was simple: use sudo bpftool map dump id {map id} to check the number of entries in the eBPF map. After nginx reload, once all shutting down worker processes exited, the map entry count again matched the worker count. No leak.
But then I had a new thought. The default nginx eBPF map size is four times the worker process count. If nginx is repeatedly hot-upgraded while old processes have not exited, the map size will not be enough. I easily reproduced this edge case by deliberately keeping several idle requests so old processes could not exit. But I do not think this needs fixing, because repeated hot upgrade is an unreasonable usage scenario.
Second, why does nginx use BPF_MAP_TYPE_SOCKHASH rather than BPF_MAP_TYPE_REUSEPORT_SOCKARRAY? I think BPF_MAP_TYPE_SOCKHASH is more flexible. It is a real hash table. As long as the key is stable, the selected sk can stay unique and unchanged. BPF_MAP_TYPE_REUSEPORT_SOCKARRAY uses an array, with the key as an array index. Adding and deleting entries requires much more care to avoid breaking the mapping relationship, and some complex scenarios become less flexible.
Finally, what key should each Reuseport sk use in BPF_MAP_TYPE_SOCKHASH? How can every sk have a unique key? nginx source code gives a general pattern here. Every socket has a unique cookie provided by the kernel. nginx uses this cookie as the eBPF map key, then includes it in the QUIC connection ID. This ensures the QUIC connection is always received by the socket corresponding to that cookie.
Does It Fail to Handle Hot Upgrade Correctly?
Then I carefully read nginx's eBPF code: src/event/quic/bpf/ngx_quic_reuseport_helper.c. While reading, I suddenly found a new problem: during nginx hot upgrade, new requests may be assigned to old workers.
As shown in the diagram, when nginx hot upgrade happens, if old processes still have active requests, the UDP sockets held by old processes are not closed. The eBPF map contains sockets from both old and new processes. QUIC connections already being served by old or new processes can correctly query their own sk and be matched correctly.
But there is a case: new requests definitely cannot hit the eBPF map. So they can only follow logic 2 in the diagram, using UDP four-tuple load-balancing hash lookup. The Reuseport socket group contains sockets from both old and new processes, which means some requests will be assigned to old processes. At that time, old processes have already entered ngx_exiting state. From the nginx code, those requests can only detect failure through timeout.
Trying to Reproduce the Problem
After realizing nginx might have this issue, I found it unbelievable. It was hard to imagine nginx making such a mistake. So I immediately tried to reproduce it.
Start by Adding Logs
I expected reproducing the issue to be simple. Enable nginx eBPF, keep old workers alive during hot upgrade with some long-lived requests, then send new HTTP/3 requests to nginx. I expected some requests to fail. But every request sent by curl successfully got a response. That was awkward. It did not match my source-code analysis. Did I misunderstand something?
To analyze further, I had a sudden idea: why not add logs to the nginx eBPF program so I can see things more clearly? Then I stepped into a pit. The real topic of this post finally begins.
First, I tried adding more detailed logs to src/event/quic/bpf/ngx_quic_reuseport_helper.c. Without thinking much, I changed debugmsg to the eBPF logging function bpf_trace_printk. Of course, the second argument, the length of fmt, also needed to be supplied. Then I tried using the build scripts under src/event/quic/bpf to generate src/event/quic/ngx_event_quic_bpf_code.c.
After a few short attempts, the build succeeded, but I hit a compile error while compiling the nginx eBPF program: error: variable has incomplete type 'struct bpf_map_def'. I did not think too much at the time. I searched the web and found Explain where bpf_map_def comes from. The discussion was long, and I did not have the patience to read it carefully, so I temporarily added a full definition of struct bpf_map_def myself. I did not think about why the libbpf header did not include it. But I was confused: the eBPF map variable ngx_quic_sockmap was declared but not defined. Why did nginx not implement this map definition?
After handling that, I directly replaced the generated ngx_event_quic_bpf_code.c. But when starting nginx, eBPF bytecode loading failed with cannot call GPL-restricted function from non-GPL compatible program. This error was easy to understand. bpf_trace_printk requires the eBPF program to follow the GPL license, while nginx uses a BSD-like license. For debugging, we need to manually change the License code to GPL, so the bpf system call sees it when loading the eBPF bytecode. I am still curious why bpf_trace_printk requires GPL. Perhaps because it has poor performance and is only for debugging, so the kernel only wants GPL-compatible programs to use it.
Then I hit another bytecode loading error. I pasted part of the error below. It left me completely lost. Looking back at my changes, aside from map definition, I only changed one debugmsg line to bpf_trace_printk("nginx quic socket selected by key 0x%llx", sizeof("nginx quic socket selected by key 0x%llx"), key);. I could not understand why it failed.
132: (bf) r3 = r2 ; R2_w=pkt(off=13,r=14) R3_w=pkt(off=13,r=14)
133: (07) r3 += 9 ; R3_w=pkt(off=22,r=14)
134: (2d) if r3 > r5 goto pc-6 ; R3_w=pkt(off=22,r=22) R5=pkt_end()
135: (71) r4 = *(u8 *)(r2 +1) ; R2_w=pkt(off=13,r=22) R4_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,var_off=(0x0; 0xff))
136: (67) r4 <<= 56 ; R4_w=scalar(smax=0x7f00000000000000,umax=0xff00000000000000,smin32=0,smax32=umax32=0,var_off=(0x0; 0xff00000000000000))
137: (71) r3 = *(u8 *)(r2 +2) ; R2_w=pkt(off=13,r=22) R3_w=scalar(smin=smin32=0,smax=umax=smax32=umax32=255,v
2025/08/20 10:55:59 [emerg] 3605800#3605800: ngx_quic_bpf_module failed to initialize, check limits
Debugging the Logging Problem
I quickly reread parts of learning-ebpf. Learning while practicing. I suddenly thought: if the eBPF bytecode has a problem, why not try loading it directly with bpftool and see whether there are clues?
So I ran sudo bpftool prog load ngx_quic_reuseport_helper.o /sys/fs/bpf/reuseport. It failed again: libbpf: elf: legacy map definitions in 'maps' section are not supported by libbpf v1.0+. I looked it up and found that libbpf had dropped support for old-style struct bpf_map_def SEC("map") my_map;. I quickly switched to the modern style: define the full map structure and put it in the .maps section. But I now had another question: what is the benefit of this new style? We will come back to that.
After changing it, I hit another load error: libbpf: BTF is required, but is missing or corrupted. This one was not hard. I quickly learned that BTF is type information for eBPF, somewhat like DWARF debug information. With clang, adding -g can generate it. But I became confused again. Debug information should not affect eBPF bytecode loading in theory. Why is BTF required here, and why does nginx not need it?
Next error: libbpf: failed to guess program type from ELF section 'ngx_quic_reuseport_helper'. From the message, I realized the ELF section name has rules. bpftool infers the eBPF program type from the ELF section name. So I changed the corresponding code to sk_select_reuseport.
Finally, bpftool succeeded. The eBPF bytecode loaded successfully. But this was not good news, because my question remained: why did nginx fail to load the same eBPF bytecode while bpftool succeeded? That made no sense.
I quickly used strace to compare nginx and bpftool when loading bytecode. I ran sudo strace -e bpf bpftool prog load ngx_quic_reuseport_helper.o /sys/fs/bpf/reuseport, then compared the bpf calls. I found that bpftool created one more eBPF map than nginx. sudo bpftool map show showed the extra map named .rodata.str1.1. I dumped it with sudo bpftool map dump id {map id}, then used readelf to inspect the .rodata.str1.1 section in the eBPF relocatable object. The contents were exactly the same.
At this point, the truth was basically clear. bpftool obviously did extra work, so the same eBPF bytecode could load successfully. That work should be eBPF bytecode linking and loading.
The log code I added used a string constant: the fmt parameter of bpf_trace_printk. This kind of string constant is not stored on the stack or heap. During compilation, clang stores it in a section such as rodata, just like normal C/C++ programs. In C/C++, the linker relocates code that accesses string constants. eBPF loading needs to do something similar. But eBPF does not have a virtual address space like a normal user-space process, so it must store the .rodata.str1.1 section in an eBPF map, then actively modify the eBPF bytecode, doing work similar to address relocation.
nginx eBPF Relocation Implementation Details
Now let's look at how eBPF relocation works. The nginx build script already gives some hints. We can print relocation details as shown below. There are two relocations in this eBPF program. One is the original nginx use of the eBPF map. The other is our new bpf_trace_printk call, which needs the call argument to be relocated to the corresponding section.
For nginx's existing approach, OFFSET in the relocation entry is the offset of the eBPF bytecode to be relocated in the .text section. Note that this offset is in bytes, and one eBPF instruction is 8 bytes. So 0x500 / 8 = 160, meaning the 161st eBPF instruction needs relocation.
/usr/bin/llvm-objdump-18 -r ngx_quic_reuseport_helper.o
ngx_quic_reuseport_helper.o: file format elf64-bpf
RELOCATION RECORDS FOR [sk_reuseport]:
OFFSET TYPE VALUE
0000000000000500 R_BPF_64_64 ngx_quic_sockmap
0000000000000560 R_BPF_64_64 .rodata.str1.1
In the nginx source, nginx modifies that instruction in two places. I first found the instruction through objdump -d: line 160: 18 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 r2 = 0x0 ll. One modification sets src_reg to 1. The other sets imm to the newly created eBPF map fd.
To understand what these changes mean, I read the eBPF instruction set document. From objdump's friendly output r2 = 0x0ll, I knew this instruction assigns to register R2. The instruction set says R2 represents the second argument to an eBPF function call. The second argument of bpf_sk_select_reuseport is exactly the eBPF map, so we were in the right place.
Then I decoded the opcode according to the instruction set document: BPF_LD | BPF_IMM | BPF_DW, meaning load a 64-bit immediate value into R2. So this is actually two instructions, because the immediate value is 64 bits. No wonder the relocation type is R_BPF_64_64.
Now nginx's two operations make sense. First, set the source register to 1. The instruction set document explains that if we want to refer to a map by eBPF map fd, set it to 1 (dst = map_by_fd(imm)). nginx's operation is very clear. Also, fds are 32-bit, so nginx only needs to modify the immediate value in the first instruction.
Solving the Logging Problem
At this point, the logging problem is easy to solve. There are two approaches.
The first is already provided by nginx. In this code, nginx stores the fmt string in a stack variable. Then no constant string is needed, and no relocation is needed. Logs work directly. I only noticed this after coming back to the code, and I almost slapped my own thigh. libbpf also has a logging macro, bpf_printk, that does similar work and more.
The second approach is to implement relocation for the fmt string ourselves. In terms of efficiency, the first approach is better. But the second is more general, especially because eBPF stack space is strictly limited. The work is essentially repeating what nginx does: create an eBPF map, find relocation information from the rel section, find the corresponding instruction, and apply relocation according to the instruction documentation. I provide a C example later.
Finally, directly using calls such as bpf_printk for eBPF logging is not recommended. This blog post analyzes the reasons in detail. The main issues are performance and lack of flexibility. In practice, eBPF logging can be classified as sending data from eBPF space to user space. Production implementations usually use BPF_MAP_TYPE_PERF_EVENT_ARRAY or BPF_MAP_TYPE_RINGBUF to pass log data to user-space processes for display and analysis. Also, I recommend the aya-log project. It is elegant and a good example of this kind of implementation.
How Loaders Handle eBPF Map Definitions
There are still unanswered questions. The one I wanted to understand most was: why did bpftool remove support for the old eBPF map definition style, and why does the new interface require debug information to load eBPF maps correctly?
First, I understood why nginx does not define eBPF map details inside the eBPF program, such as map type and key/value types. nginx implements a simple special-purpose eBPF bytecode loader, and the map definition is hardcoded in nginx startup code. It does not need to be defined in the eBPF program.
So I only needed to study how eBPF loading tools or libraries process map definitions. The method is simple: look at where map definition information is stored and how it is consumed. Since I have been learning Rust, I will use aya-obj as the example.
In the old struct bpf_map_def style, map definitions are placed in the maps section through the SEC macro. The loader only needs to read that section to know what eBPF maps to create. In aya source, it reads the maps section and parses it according to the symbol table. Then this code parses each struct bpf_map_def definition. The loader then knows what map type to create and what key/value sizes to use.
The downside is that value_size and key_size in bpf_map_def only store the sizes of the key and value types. That means the loader cannot do type checking. If key/value types do not match when the map is created or used, the compiler cannot warn the user. Even the eBPF verifier cannot report this, because it only checks memory usage. If the map defines a 4-byte key type but the program passes an 8-byte key, it may simply be truncated, especially when user space performs map CRUD operations. To verify this, I tested it. The incorrect eBPF bytecode loaded successfully. clang and the eBPF verifier stayed completely silent.
The newer map definition below solves this problem. First, section naming tells the loader how to parse the map definition. .maps is the new map definition section. Second, the __type macro uses real type definitions for key and value, reducing the issue above.
But how does the loader know the map definition details? Normally, DWARF debug information contains concrete definitions of struct members. eBPF goes further and defines BTF based on DWARF. BTF is more compact and mainly keeps type and field layout, which is enough to represent map definitions. eBPF loaders rely on BTF to identify details of new map definitions. See aya-obj source for an example. BTF has a more important role too: CO-RE. I will not expand on that here. If I gather enough interesting material someday, I may write another post.
struct {
__uint(type, BPF_MAP_TYPE_SOCKHASH);
__uint(max_entries, 1024);
__type(key, __u64);
__type(value, __u64);
} ngx_quic_sockmap SEC(".maps");
At the time, I could not resist writing a simple eBPF loader in C. It loads the simplest bpf_trace_printk eBPF program onto the Reuseport hook. If reproducing nginx's code is too troublesome, you can refer to my code.
Continuing to Debug the nginx Hot Upgrade Problem
At this point, I could easily add all the eBPF logs I cared about. I tried reproducing the issue again by sending HTTP/3 requests with curl to nginx during hot upgrade. From the eBPF logs below, we can see that QUIC handshake packets for new curl requests did not hit the nginx eBPF map. They fell back to the kernel's default four-tuple hash matching. Since the old process's Reuseport UDP socket had not closed, it also had a chance to process the packets.
sudo cat /sys/kernel/debug/tracing/trace_pipe
bpf_trace_printk: nginx quic default route for key 0xbcf02d56a2dd29ef
bpf_trace_printk: nginx quic default route for key 0xbcf02d56a2dd29ef
bpf_trace_printk: nginx quic default route for key 0x21bddac2ce121be5
bpf_trace_printk: nginx quic default route for key 0x21bddac2ce121be5
Looking at the packet capture, curl sent two QUIC handshake packets. This was not retransmission. The handshake data was too large to fit in the default 1200 bytes. From the frame_type column, the first handshake packet has value 6, carrying only QUIC crypto frame, while the second has 6,0, also carrying QUIC Padding frame. So the two handshake packets are different.
Rows three and four are nginx responses to the first two handshake packets. From long.packet_type value 3, we know they are QUIC Retry packets. Since QUIC Retry was enabled in nginx config, this is expected. From nginx logs, I confirmed that the new curl request was handled by an old nginx process and rejected. Then why did curl show the request succeeded?
sudo tshark -i any -Y quic -T fields \
-e frame.number -e frame.len \
-e ip.src -e udp.srcport \
-e ip.dst -e udp.dstport \
-e quic.dcid \
-e quic.frame_type -e quic.long.packet_type
286 1244 127.0.0.1 56477 127.0.0.1 1984 bcf02d56a2dd29ef 6 0
287 1244 127.0.0.1 56477 127.0.0.1 1984 bcf02d56a2dd29ef 6,0 0
288 153 127.0.0.1 1984 127.0.0.1 56477 3
289 153 127.0.0.1 1984 127.0.0.1 56477 3
290 1244 127.0.0.1 56477 127.0.0.1 1984 21bddac2ce121be5f1644d62b3183f08c8286ebc 6 0
291 1244 127.0.0.1 56477 127.0.0.1 1984 21bddac2ce121be5f1644d62b3183f08c8286ebc 6,0 0
292 165 127.0.0.1 1984 127.0.0.1 56477 3
293 165 127.0.0.1 1984 127.0.0.1 56477 3
I went back to the curl output and saw a huge HTTP/2 string. So curl had quickly fallen back and retried. I should have used --http3-only. Maximum embarrassment.
./curl -v -i -sS -k --http3 --connect-timeout 30000 --max-time 300000 https://127.0.0.1:1984/t
* Trying 127.0.0.1:1984...
* Trying 127.0.0.1:1984...
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519MLKEM768 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
* subject: CN=localhost
* start date: Nov 5 06:54:15 2021 GMT
* expire date: Jan 4 06:54:15 2041 GMT
* issuer: CN=localhost
* SSL certificate verify result: self-signed certificate (18), continuing anyway.
* Certificate level 0: Public key type RSA (4096/152 Bits/secBits), signed using sha256WithRSAEncryption
* Connected to 127.0.0.1 (127.0.0.1) port 1984
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://127.0.0.1:1984/t
* [HTTP/2] [1] [:method: GET]
Reporting the Issue to the Community
Since nginx's eBPF implementation had a flaw, hot upgrade could cause a significant portion of new requests to fail. I sent an email to the nginx development group. At the time, nginx had not yet moved to GitHub. This is the email. I mainly explained the severity. At least in my eyes, it was not acceptable. If quic_retry was not enabled, many new requests could only retry after timeout. In a demanding production environment, I would call that an incident.
Soon, a senior developer replied. The name looked familiar. After checking, I realized he was the author of ngx-rtmp. I was immediately excited. A huge amount of live-streaming traffic in China has run through ngx-rtmp, and the module is very well written. I had learned a lot from it. As expected from a strong engineer, he immediately sent two patches: one eBPF-based optimization, and one Established-over-unconnected approach that I had mentioned in the previous post.
I was also happy to share the core problem Established-over-unconnected hits on Linux: hash-table lookup performance degradation. He summarized it perfectly: Apparently current UDP stack is just not made for client sockets. He also pointed out one pain point of eBPF, namely that it requires special permissions to run. Personally, I think that is acceptable and not my main pain point.
How to Fix It
Now let's analyze the eBPF optimization patch. I think the fix itself is simple. The essence of the problem is: when old and new nginx processes coexist, with the current nginx eBPF implementation, new requests are destined to miss the eBPF map. A new request can only hit the map after the handshake negotiates a new Connection ID and puts the correct socket cookie into it. So new requests must be hashed by UDP four-tuple and follow the kernel Reuseport Group lookup logic.
The solution is clear. Since the kernel default logic may assign new requests to old processes, we need to handle new requests inside the eBPF program and prevent them from reaching the kernel default lookup logic. We can maintain another map in eBPF specifically for new requests, effectively reimplementing the kernel Reuseport Group hash lookup logic inside eBPF. Then, during nginx hot upgrade, when old and new processes coexist, we can simply remove old-process sockets responsible for new requests from this new map.
In Roman's patch, UDP sockets are split into two sets. One set is stored in the ngx_quic_listen map and handles new requests. The other set is stored in the ngx_quic_worker map and handles established requests. During hot upgrade, old processes delete their sockets from ngx_quic_listen, so they no longer receive new requests. Their sockets remain in ngx_quic_worker, so old requests are unaffected.
Everything looks good. The only frustrating thing is that as of August 21, 2025, this eBPF fix patch still has not been merged into nginx mainline.
Recommended nginx HTTP/3 Configuration Today
For nginx 1.27.2 and earlier, or more broadly before 1.29.1, though I have not tested the newest versions and the code looks similar, here are the configuration points:
reuseportmust be enabled in thelistenoptions. Without it, QUIC request serving may break in multi-process mode.quic_ebpf on;is recommended. Without it, QUIC connection migration has problems.During hot upgrade, old requests may also be passively cut off.Additional note: current nginx implementation destroys existing HTTP/3 requests during hot upgrade because they are marked idle. The logic is here and here. I do not know when this will be optimized.quic_retry on;is recommended. Without it, during hot upgrade, nginx old workers may mishandle new requests. Additional note: this adds one RTT to normal QUIC handshakes, but for public servers it is quite necessary. I analyzed that trade-off in the QUIC Retry section of an earlier post.
Retrospective
The sections above reviewed my process of learning nginx's eBPF implementation two years ago. I spent a lot of time trying things back and forth before finding the answer. Now I want to review the process and the mistakes I made.
First, the root cause of my initial misjudgment was curl's fallback retry mechanism. It prevented me from reproducing the expected problem. I should reflect on that. If I had carefully looked at curl output, nginx error logs, or packet captures, I would not have made that mistake.
Second, I spent too much time on the eBPF log loading failure. Normally, when debugging problem A and a derived problem B appears, you should evaluate whether solving B is worth the cost and whether B is required to solve A. Losing focus without noticing is dangerous. Many times, we need to step out of the detail swamp and look at the whole problem. In this case, the whole investigation was my own learning exercise with no deadline pressure, so the distraction was fine. Understanding the problems step by step was the source of the fun.
Finally, although I spent a lot of time, I do not think it was wasted. For learning, fast can be slow, and slow can be fast. Nearly two years later, I can still easily reconstruct the pitfalls I hit, write a long post about them, and use them as a foundation for later eBPF learning. Those efforts were not wasted. I have successfully convinced myself.
udpgrm Implementation Analysis
At this point, I should probably stop. This post is already bloated. But I still want to keep the momentum and share what I learned from Cloudflare's newly open-sourced project udpgrm. This project uses eBPF to solve the UDP hot upgrade problem I have been discussing. Let's see how Cloudflare handles it.
How It Is Exposed to Developers
I learned about this project around May this year. My first reaction was surprise. I did not expect this scenario could be turned into a general solution. Before reading the code, I first thought about what problems a general solution would need to solve and how I might solve them. This is one of my favorite parts of reading open-source projects.
- UDP-based protocols are diverse, and their connection identifiers differ. How can eBPF identify which connection a UDP packet belongs to, and then decide which socket in the eBPF map should handle it?
- How does udpgrm manage Reuseport UDP sockets? Are all sockets managed solely by udpgrm? How does udpgrm integrate with various runtimes? How does udpgrm identify whether a Reuseport socket is managed by itself?
For the first question, I could not think of a good solution. Would it define a unified standard, such as standardizing the QUIC Connection ID format? Ignoring security considerations, the immediate problem is that users would have to heavily modify their QUIC stack implementation. What about other UDP-based protocols?
I quickly read the udpgrm documentation and code. I found that udpgrm implements a cBPF interpreter inside eBPF. That solves the problem. Developers can write cBPF bytecode themselves and freely read their cookie value from UDP packets, achieving an effect similar to nginx's eBPF program.
When I found the cBPF interpreter, I was a little shocked. Maybe I simply had not seen enough, but I really did not expect this solution. Providing flexibility through programmability is always viable. I looked at the implementation out of curiosity. An interpreter is definitely a large loop. eBPF is strict about loops. udpgrm uses bpf_loop to implement the loop. So eBPF already supports bounded loops.
For the second question, I also did not think of a great solution. I guessed it might need an extra eBPF map to record socket cookies managed by udpgrm. But how does udpgrm mark that a Reuseport UDP socket belongs to it? Maybe by Reuseport Group: as long as the socket is in the group, it is considered one of its own.
Reading the implementation, I found that udpgrm uses BPF_PROG_TYPE_CGROUP_SOCKOPT directly in getsockopt or setsockopt and defines custom option values, such as UDP_GRM_WORKING_GEN. Users can explicitly configure udpgrm behavior on their sockets through setsockopt. udpgrm eBPF also uses bpf_set_retval to ensure custom option values do not affect the original system calls, while handling many edge cases.
This was the first time I learned such an operation was possible. Maybe projects like Cilium have done this for a long time, since the relevant eBPF support appeared early. I had simply not seen enough. But learning it now is still not too late. I may use it in my own projects someday.
Besides requiring applications to adapt to udpgrm's activation interface, udpgrm also provides udpgrm_activate.py to reduce user effort. Users configure udpgrm features through script parameters. The Python script creates sockets and sets udpgrm options through setsockopt. Then comes the familiar parent-child socket inheritance flow. udpgrm_activate.py starts the user's application through fork+exec. The user's server already holds the sockets created by the script.
How does the user's server know which file descriptors those sockets use? There are many ways. In udpgrm's sample program, it simply tries to open file descriptors in the range [3, 3+32]. That works but is not precise. nginx gives us another example: pass file descriptors through environment variables between parent and child processes. So the user's server still needs some adaptation, but the work becomes simple.
udpgrm Design and Usage Details
Work Generation
udpgrm introduces the concept of work generation. It is easy to understand because udpgrm solves UDP hot upgrade. The Reuseport UDP socket group held by the current application service and the group held by the new service after hot upgrade are the same Reuseport Group at the kernel level, but they are different work generation values for the udpgrm eBPF program.
Using nginx hot upgrade as an example, Reuseport UDP sockets held by shutting-down workers belong to the previous work generation, while sockets held by newly created workers naturally belong to the new generation. If this is unclear, refer to the diagram in the linked documentation. I originally wanted to draw one here, but the documentation's diagram is already excellent, so I will not add a worse one.
How should users create work generation, especially sockets from different generations? It is simple. Each socket can set its own work generation id through the UDP_GRM_WORKING_GEN option in setsockopt, and can also retrieve the current generation id through getsockopt.
The setsockopt eBPF program sends registration information to the udpgrm background daemon through ringbuf. The daemon has dedicated logic to update the socket into the corresponding eBPF map. Then udpgrm's BPF_PROG_TYPE_SK_REUSEPORT program handles UDP packets and dispatches them to the correct work generation. If one generation has multiple sockets, it also needs load balancing.
udpgrm Modes
Now let's discuss several udpgrm modes.
DISSECTOR_FLOW Mode
I personally do not like this mode much because it has several drawbacks.
The first is that it uses UDP four-tuple information as the hash-table lookup key. That binds the UDP-based protocol to the UDP four-tuple. It also means high concurrency is hard to support, because eBPF map sizes have upper limits, a topic we have discussed before.
The second drawback is that it uses a 3-tuple hash: {remote IP, remote port, reuseport group ID}. I feel the collision probability is much higher. It does not use local IP and port because the sendmsg eBPF program cannot see that information.
Why use a sendmsg eBPF program? Because entries in the eBPF map need an eviction mechanism.
udpgrm uses a BPF_MAP_TYPE_LRU_HASH map here, so if the map is full, it updates according to LRU semantics. But a connection lifetime timeout mechanism is still better, because it spreads out the cleanup cost. That lifetime mechanism depends on whether the corresponding connection sends UDP packets, and it is maintained inside the sendmsg eBPF program.
The sendmsg program also inserts into the LRU map. In other words, udpgrm records a flow only after the server replies to the client's handshake packet. This reduces unnecessary overhead and lowers pressure on the LRU map.
The collision problem is clearly described in the udpgrm documentation. The LRU map key is 4 bytes, and the documentation says the collision probability is 1 / 524288. This is easy to understand: the probability that the 3-tuple hash generates the same 4-byte value is 1 / 2^32. With 8192 concurrent flows, collision probability is 8192 / 2^32, which is 1 / 524288.
The consequence is that a new request may be mistaken for an old request and assigned to an old service. So the documentation repeatedly emphasizes: an application should be capable of accepting new flows even in draining mode. If the application supports this, the old service just exits more slowly. Unfortunately, many services may not support it. For example, nginx shutting-down workers do not continue processing new HTTP/3 requests.
Does this mode have advantages? Yes. It is simple and easy to use. It does not require extra development in the UDP-based transport protocol, such as putting socket cookies into the connection identifier, and it does not require writing annoying cBPF bytecode to extract cookies from UDP payload.
DISSECTOR_CBPF Mode
Now let's look at the cBPF mode. There is not much more to say about the implementation details. After reading the code, I found that aside from cBPF, the rest is very similar to the nginx eBPF optimization patch. Both ensure all requests are handled by the Reuseport eBPF program and do not fall back to the kernel's default four-tuple hash distribution.
One interesting detail is that the custom cBPF bytecode only needs to read 2 bytes, while a socket cookie is 8 bytes. udpgrm maintains an extra two-dimensional u64 array that stores cookies. The 2-byte data contains the specific index in that array. The eBPF program can quickly use the index to find the socket cookie. Once it has the cookie, it can find which socket should serve the flow.
Those 2 bytes contain a lot of information. The parsing logic is here. It stores not only work generation id, but also the index in the cookie array for that specific generation. More importantly, it uses a checksum to confirm whether these 2 bytes conform to udpgrm's custom format. If not, the packet is basically a new flow.
Then we need to look at how new flows are handled, because this is key to avoiding nginx's current eBPF mistake: assigning new requests to old services. The udpgrm code checks the current latest work generation id, how many sockets the current generation has, and then uses UDP four-tuple hash load balancing to choose a suitable socket inside the current generation.
Honestly, I think udpgrm's DISSECTOR_CBPF mode is even better than nginx's eBPF optimization patch. The main reason is the work generation concept. Once introduced, many things become clear. Strong engineers really are different.
Application Selection
udpgrm also provides a more fine-grained dimension: application selection. This is very cool. I had never thought about this scenario, though I will not belittle myself too much, because it clearly looks like a product of specific business requirements.
More concretely, on the same machine, behind the same listening address and port, there may be multiple application services. If the eBPF program can distinguish which application a UDP datagram belongs to and assign it to the socket held by that application, that is excellent.
How do we decide which application a UDP packet belongs to? The application must add its own information into the UDP packet and write cBPF bytecode to parse it. udpgrm gives an example, DISSECTOR_BESPOKE, using the SNI extension in the TLS ClientHello to determine the application. This is the familiar request domain name.
The example is specifically written to extract TLS SNI from QUIC handshake packets. We all know QUIC integrates TLS 1.3, but I was still surprised to see QUIC handshake packets parsed inside an eBPF program. It feels like anything can be put into eBPF. Very cool.
Still, I want to nitpick. This QUIC parser only works on QUIC Initial packets. Of course, it cannot decrypt packets in other spaces. But the parsing logic assumes that the first UDP datagram in the QUIC Initial space carries the TLS SNI extension. If my ClientHello is large enough and contains many extensions, causing the QUIC handshake to be sent across two UDP datagrams, and TLS SNI is in the second datagram, then this parser should not work. It has no buffer to handle the reliable Crypto byte stream. In production, this is unlikely, especially because QUIC Initial packets must be padded to at least 1200 bytes. Unless someone deliberately creates trouble, this should not happen.
Building a Whole System in a Tiny Box
I previously read code from parca-agent, which implements user-space stack unwinding in eBPF and even supports some virtual-machine languages such as Python and Ruby. This time, in udpgrm, I saw a cBPF interpreter and QUIC handshake parsing, including AES128 and SHA256 algorithms, implemented in eBPF. This reminded me again how much potential eBPF has. Good projects keep appearing, and there is too much to learn. Fitting so much logic into such a constrained execution environment takes real skill and courage.
That is all for my analysis of udpgrm. It may not be deep enough, but since the project is new and not widely known, this post may be one of the few Chinese-language analyses of it. I spent only a few nights reading udpgrm source code, and I personally gained a lot. If I misunderstood anything, please point it out. Later, I may only return to udpgrm when I have a chance to work on a related project, either to borrow ideas from it or, more politely, to learn from it.
Epilogue
This post is also the end of a hidden mini-series, and of course the start of my eBPF self-learning series. The hidden mini-series mainly discussed pain points in UDP network programming on Linux, including the previous two posts: Implementing QUIC from Scratch with Rust: Connection Migration and Linux Server-Side UDP Network Programming: Zero-Loss Hot Upgrade.
Good articles should balance detail and brevity. These posts are admittedly bloated, tightly connected, and require a lot of background knowledge. But I wrote them this way, and I am still satisfied with them. They are a complete summary of pitfalls I hit years ago. For me, being able to write down the remaining memories in my head with some logic already counts as success.