Certainly! Here’s the rewritten content:Author: Yaxiong ZhaoThis document provides insights into HTTP/2 observability
In today’s world filled with microservices, obtaining observability over the messages exchanged between services is crucial for understanding and troubleshooting problems.
Unfortunately, the dedicated header compression algorithm of HTTP/2, known as HPACK, makes tracking HTTP/2 complex. While HPACK contributes to the efficiency of HTTP/2 over HTTP/1, its stateful algorithm sometimes renders typical network tracing tools ineffective. This means that tools like Wireshark cannot always decode plaintext HTTP/2 headers from network traffic.
Fortunately, by using an eBPF uprobe, you can trace the traffic before it gets compressed, allowing you to debug your HTTP/2 (or gRPC) applications.
This article will answer the following questions
and share a demonstration project on how to trace HTTP/2 messages using an ebpf uprobe.
Wireshark is a well-known network sniffing tool that can capture HTTP/2. However, Wireshark can sometimes fail to decode HTTP/2 headers. Let’s see it in practical application.
If we start Wireshark before launching the gRPC demo, we can see the captured HTTP/2 messages in Wireshark:
Wireshark captured the HTTP/2 header frames.
Let’s focus, it’s equivalent to headers in HTTP 1., recording the metadata of an HTTP/2 session. We can see a specific HTTP/2 header block fragment with raw bytes bfbe. In this case, the raw bytes encode the grpc-status and grpc-message headers. Wireshark decodes them correctly as follows:
If Wireshark is started before the message stream begins, it can decode HTTP/2 HEADERS.
Next, after starting the gRPC client and server, let’s start Wireshark. The same messages are captured, but the raw bytes are no longer decoded by Wireshark:
After the message stream starts, Wireshark cannot decode HTTP/2 HEADERS.
Here, we can see the Header Block Fragment still shows the same raw bytes, but the plaintext headers cannot be decoded.
To replicate this experiment yourself, follow the instructions.
If Wireshark is started after our gRPC application begins transmitting messages, why can’t it decode HTTP/2 headers?
This is because HTTP/2 uses to encode and decode headers, compressing headers,.
HPACK works by maintaining the same lookup tables on both the server and the client. In these lookup tables, headers and/or their values are replaced by their indices. Since most headers are transmitted repeatedly, they are replaced by indices that consume significantly fewer bytes than plaintext headers. Thus, the network bandwidth used by HPACK is considerably reduced. Since multiple HTTP/2 sessions can be multiplexed over the same connection, this effect is amplified.
The diagram below illustrates the tables maintained by the client and server for response headers. New header name and value pairs are appended to the table, replacing old entries if the lookup table reaches its size limit. When encoding, plaintext headers are replaced by their indices in the table. For more information, see.
HTTP/2’s HPACK compression algorithm requires the client and server to maintain the same lookup tables to decode headers. This makes it difficult for tracers that do not have access to this state to decode HTTP/2 headers.
With this knowledge, the results of the above Wireshark experiment become clear. When Wireshark runs before the application start-up, it records the entire history of headers, allowing Wireshark to regenerate precisely the same header table.
When Wireshark starts after the application runs, it loses the initial HTTP/2 frames, resulting in encoded bytes bebf lacking corresponding entries in the lookup table. Consequently, Wireshark cannot decode the corresponding headers.
HTTP/2 headers are the metadata of an HTTP/2 connection. These headers are critical information for debugging microservices. For instance, :path contains the requested resource; content-type is needed to detect gRPC messages and then apply protobuf parsing; and gRPC-status is necessary to determine the success of a gRPC call. Without this information, HTTP/2 tracing loses much of its value.
If we cannot properly decode HTTP/2 traffic without knowledge of the state, what do we do?
Fortunately, eBPF technology allows us to gather the information we need by probing the HTTP/2 implementation without requiring state.
Specifically, eBPF uprobe addresses the HPACK issue by directly tracing plaintext data within the application’s memory. By attaching a uprobe to the API of the HTTP/2 library that accepts plaintext header information as input, the uprobe can read header information directly from the app’s memory before HPACK compression.
demonstrates how to implement a uprobe tracing program for HTTP applications written in Golang. The first step is to identify the function to which BPF probes will attach. The function’s parameters should contain the information of interest. Ideally, the parameters should have a simple structure for easy access in BPF code (through manual pointer tracing). Additionally, the function should be stable to ensure the probe applies across various versions.
By examining the source code of Golang’s gRPC library, we determine that loopyWriter.writeHeader() is an ideal tracing point. This function accepts plaintext header fields and sends them to an internal buffer. The function signature and type definition of its arguments have remained stable since.
The current challenge is identifying the memory layout of the data structure and writing BPF code to read data at the correct memory addresses.
Let’s take a look at this function’s signature:
The task is to read the contents of the 3rd argument, hf, which is a slice of HeaderField. We use the dlv debugger to compute offsets of nested data elements, with results shown in.
This code performs three tasks:
Let’s run the uprobe HTTP/2 tracing program and then start the gRPC client and server. Note that this tracer works even when started after establishing a connection between the gRPC client and server.
Now we see the response headers sent from the gRPC server to the client:
We also implemented a probe in probe_http2_server_operate_headers() for google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders(), tracing headers received on the gRPC server.
This allows us to observe the request headers received by the gRPC server from the client:
Uprobe-based tracing programs for production environments require further considerations, which you can read about in the footnotes. To try out this demonstration, see the instructions here.
HPACK header compression makes tracing HTTP/2 traffic challenging. This article demonstrates an alternative way to capture messages using an eBPF uprobe to directly trace the appropriate function within an HTTP/2 library.
It’s crucial to understand that this approach has pros and cons. Its primary advantage is the ability to trace messages regardless of when the tracer is deployed. However, a significant drawback is the specificity to a single HTTP/2 library (in this case, Golang’s library); for other libraries, this exercise must be repeated and may need maintenance if upstream code changes. In the future, we’re considering providing USDT for libraries, which would offer more stable tracing points and alleviate some drawbacks of uprobe. Ultimately, our goal is to optimize a turnkey method without regard for deployment order, which is leading us to adopt a root-based eBPF strategy.
Looking for the demonstration code? Find it.
Questions? Find us on or Twitter.