Mastering Packet Capturing at the HTTP Application Layer for Effective Debugging

Introduction

Packet capturing at the HTTP application layer has become an essential part of everyday testing and debugging. Recently, I started working on a new project and suddenly realized that all my previous capturing methods were ineffective. The interaction between modules, both front-end and services, became invisible, leaving me feeling blindfolded.

In fact, I noticed early on that apps like Alipay and others couldn’t capture requests using proxy software like Fiddler or Charles by default. However, capturing software like Wireshark could monitor the traffic of these apps, indicating that their main application layer protocol is still HTTP (https). Nevertheless, our proxy packet capturing tools became ineffective. Now that I’ve encountered this in practical work, I have no choice but to resolve it, as having something obscuring my vision makes me uncomfortable.

Principle of Proxy Packet Capturing

To understand why Fiddler or Charles doesn’t work for these apps, we need to understand the principle of proxy packet capturing (of course, if you’re not interested, you can skip directly to the practical steps). Fiddler or Charles uses a proxy which is entirely different from Wireshark (Wireshark uses network card data replication, capturing any data passing through the specified network card). It only affects application layer network protocols using proxies, such as the common HTTP(s), Websocket, etc.

Let’s take HTTP as a simple example.

For an HTTP request, a client needs to locate the server, connect via TCP using the hostname and port specified in the HTTP request URL, and then send the HTTP message to the target server (for more details, see https://tools.ietf.org/html/rfc7232).

Let’s look at how an HTTP proxy operates. When you start Fiddler or Charles, you create an HTTP proxy server. These tools notify the operating system about the new proxy server, specifying its IP and port. On Linux, you can manually inform the system (export http_proxy=ip:port export https_proxy=$http_proxy). On mobile devices, you specify this in the current Wi-Fi settings. Now that the system knows you want to use a proxy, when the HTTP client sends a request, it connects to the proxy server instead of performing DNS resolution and connecting to the target server. The proxy server establishes a connection with the client, and based on the request information, connects to the actual server.

In a normal scenario without a proxy, HTTP requests sent from the client to the server include only part of the URI.

For example, in a request to www.baidu.com/index.html without a proxy, the request line is actually GET /index.html HTTP/1.1, not the full URI. This is because the original HTTP design didn’t account for intermediate servers (proxies); the client knew the server address before sending the message and established a connection, so there was no need to send the scheme, hostname, or port. However, with the introduction of proxies, this method became problematic. When a client connects to a proxy server, the proxy has no way of knowing the correct server address unless the client includes the full URI in the request, allowing the proxy to parse the real server’s address.

Since our requests are sent through a proxy server (Fiddler or Charles), the proxy capture software knows all HTTP request and response messages and can even modify requests and responses if needed.

Why Some Applications Can’t Be Captured

The key to proxy capture is requiring the HTTP client to connect to the proxy server as specified. Typically, by setting the proxy at the system level, most HTTP clients adhere to it. They check for a system proxy before making HTTP requests and use the full URI to connect to the proxy if the setting exists. However, some platforms develop their own HTTP clients, which, although they implement proxy functions as per protocol, do not necessarily use system proxies by default.

This situation is quite common in reality. The Flutter project I’m currently working on, for instance, doesn’t actively use system proxies by default, requiring separate configuration. While this policy may inconvenience testing and debugging, it also offers some data security improvement.

Since the HTTP client doesn’t use our configured system proxy, it doesn’t connect to the Fiddler or Charles proxy server, preventing us from capturing any requests.

Solutions

Since we know why Fiddler and Charles can’t capture packets and understand proxy capturing principles, solutions can be devised.

The issue arises because our app’s HTTP client doesn’t connect to the proxy server, causing our proxy capturing software to malfunction. By reconnecting the client to the proxy server (without altering client software or apps), we can resolve this.

Method 1: Control DNS resolution to make the client perceive our proxy server as the target server.

            Advantage: Convenient operation by modifying the device’s hosts file to easily implement

            Disadvantage: Hosts need pre-added for each domain; modifying hosts on devices like phones is challenging (hard to implement for mobile apps).

Method 2: Directly forward network traffic on the device by forwarding data sent to ports 80 and 443 on designated terminal devices to the proxy server’s target port.

           Advantage: Can configure device-specific settings; no additional configuration required on mobile devices.

           Disadvantage: Requires separate hardware equipment.

Method 3: Use a V** to forward device’s traffic to the proxy server.

           Advantage: Using V** software requires no additional tests.

           Disadvantage: Default V** on terminals forwards all traffic; configuring reasonably may require additional learning costs.

Practical Steps

Here, I’m directly using the third method mentioned above (considering the difficulty in operating Method 1 for phone apps and the potential need for additional devices in Method 2). Since our testing target is mobile apps, first, we install a V** on the phone. A convenient V** app is Drony (introduced here). Drony creates a V** on your phone, redirecting all traffic back to Drony itself (without streaming to a V** server), enabling it to manage all network traffic on the phone or configure individual app traffic.

1: Install Drony (Android device here)

Download the desired version of Drony online, or download from here (https://files.cnblogs.com/files/lulianqi/Drony_102.apk). After installation, open the software as shown below.

2: Activate Proxy Capture Software (Fiddler used here)

Fiddler usage isn’t detailed here; ensure remote proxy is enabled and install the Fiddler root certificate on your phone. The remote proxy address used here is 192.168.2.244:8888.

Note some certificate details: Fiddler’s default root certificate is in cer format; some phones only recognize pem format. Use OpenSSL to convert (openssl x509 -inform der -in FiddlerRoot.cer -out FiddlerRoot.pem). OpenSSL is usually installed in Windows/Mac, so you can execute the command without problems. Install the generated FiddlerRoot.pem on the phone (Charles defaults to a pem certificate).

3: Configure Drony Forwarding

Open Drony (in OFF state), navigate to the SETTING page, and select Networks Wi-Fi to configure.

Select the network the phone’s connected to (ensure the network is connected to the Fiddler proxy server network).

Configure the proxy entry for the current network (just enter the Fiddler proxy address), and select the proxy mode as manual (Manual).

Ensure to select Plain HTTP proxy as the Proxy type.

Set Filter default value to Direct all, then click below Rule to set application rules.

Your default rules should be empty; click the plus sign above to add a rule (only requests matching the rule will be forwarded). The following operations will demonstrate using apps like Xianyu or Alipay, but these aren’t the current test project apps, nor do I work for that company. They’re chosen because many people are familiar with them, and the reasons for capture failure are similar to the current project app’s.

Select the current Wi-Fi’s SSID in Network ID. Choose Local proxy chain as the Action. Select the app requiring forced proxy in Application.

Leave Hostname and Port empty, indicating everything will be forced through the proxy. Apps might use other network protocols, not just HTTP, and you may not want all traffic redirected to the HTTP proxy server, so you can specify IP and port here for forwarding.

Save upon completion, return to the SETTINGS homepage, navigate to the LOG page, and click the button below to set it to the ON state (indicating it’s enabled).

At this point, start Alipay or Xianyu, and you’ll see normal traffic in Fiddler. However, if you’re as unlucky as me, you might only see Tunnel to (TLS tunnel establishment). If you use Charles, you might see red crosses in the list.

Of course, the Fiddler root certificate was installed successfully, and Fiddler’s configuration is correct (HTTPS packet capturing in Chrome on the phone works fine).

Given that Drony has successfully directed traffic to Fiddler, the lack of the ability to decrypt HTTPS messages is likely a certificate issue. The certificate validation process will be briefly described (you can skip directly to the practical steps if not interested).

Certificate Validation Principle

Both Fiddler and Charles replace the TLS certificate as a Man-in-the-Middle attack technique, decrypting messages before encrypting and forwarding to the real server.

You can view HTTPS Man-in-the-Middle Attack Practice (Principles and Practice) for attack details.

With a proxy, the client first connects to the proxy server (the ‘attacker’ in the image). The client establishes a TLS channel with the Proxy, and naturally, the proxy has the TLS transmission key to decrypt messages.

However, due to certificate presence, the client verifies the certificate’s legitimacy before deciding to connect to a server. Installing a root certificate on use devices when using Fiddler or Charles for HTTPS capture is intended to pass client certificate verification.

In any browser, visit an HTTPS page and check its certificate information.

You’ll notice certificates include:

(1) Validity – effective period, containing the start and end times;

(2) Public Key Information – Subject Public Key Info, including encryption algorithm and contents;

(3) Fingerprint Information for verifying integrity, ensuring unmodified certificates. Fingerprint verification involves encrypting the entire certificate’s hash fingerprint [hash using CA’s private key] and including it. The client hashes the certificate using a CA’s public key in the trusted root certificate list to decrypt and verify hash matches. Mismatched hashes indicate altered content.

(4) Certificate Signature Value and Algorithm – describes hash algorithm and value used;

(5) Certificate’s Issuer CA institution;

(6) Subject, indicates the organization/company the certificate is issued to;

(7) Certificate version, Serial Number, and Extensions.

The image above depicts the fingerprint verification process, showing that verifying certificates mainly involves the CA’s public key decrypting the original fingerprint. Where does this public key come from? Devices ensure security by maintaining a trusted CA public key list (root certificate) containing authoritative entities’ public keys used when issuing certificates signed with their private keys (producing certificate fingerprints). This ensures only certificates from authoritative entities pass client verification.

Fiddler doesn’t possess certificates’ corresponding private keys (CA only provides these to website owners), preventing TLS handshake completion. To handshake, Fiddler self-signs site certificates using its private key. The client lacks CA public keys for verifycation preparations in a trusted CA list, preventing verification failure. Installing Fiddler’s root certificate on the device addresses this.

Decryption Failure Reasons

The above description clarifies that HTTPS decryption failure results from certificate verification failure, with root certificate installation incomplete.

Since Android 7.0, the system allows apps to define custom trust CA sets. Some apps only trust pre-installed system CA certificates and not user-installed CA certificates (or the app’s development framework only trusts system certificates due to developers’ inattentiveness). Android considers user-installed certificates as user certificates, thus, Fiddler or Charles’ root certificates become invalid.

Solution and Operating Method

Knowing the cause suggests solutions: installing the proxy software’s root certificate as a system certificate. Installing a certificate to the system is relatively simple by placing it in a specified location (/system/etc/security/cacerts/). First, rename the root certificate to ..

Certificate_Hash denotes the certificate file hash value, and Number is an appendix to prevent identical hash values (use 0).

Download your root certificate FiddlerRoot.cer, and use openssl x509 -subject_hash_old -in to calculate the certificate’s hash, renaming it based on the hash number (e.g., 269953fb.0 computed from my certificate’s hash; yours may vary).

Copy the 269953fb.0 file to /system/etc/security/cacerts/.

After completion, you’ll see the proxy software certificate in the system area.

Note: /system/etc/security/cacerts/ write permissions require device root access.

Copying a certificate to the directory requires rooting your device. For instructions, refer to official OEM guides. Xiaomi, for instance, provides access after 7 days of being linked to a Xiaomi account and flashing development firmware.

To verify if a Xiaomi phone supports root, refer to the Xiaomi download page.

Effect

With certificates installed in the system, revisit Fiddler for results analysis.

Re-opening Xianyu shows complete HTTPS requests. Let’s modify request return data.

Using the Fiddler plugin FreeHttp, alter response data: change “second-hand phone” to “second-hand pony” and replace images. For FreeHttp instructions, refer to the FreeHttp article.

Re-opening Xianyu, shows altered proxy data. Note: clear Xianyu’s cache and app data to ensure fresh data requests.

SecurityhttpsNetwork SecurityhttpData Analysis