1. Introduction
Packet capture at the HTTP application layer has become a crucial part of daily work, especially for testing and debugging network security. Recently, when I got involved in a new project, I suddenly found that the previous packet capture methods were no longer effective. All of a sudden, the interactions between the front-end and services between modules became invisible, and I felt like I was blindfolded. The challenge was to adapt to these changes and find a new approach to ensure effective packet capture.
In fact, I discovered a long time ago that the Alipay app and other apps I usually use cannot capture requests by default using proxy packet capture software such as Fiddler or Charles, but by using a network card packet capture software like Wireshark, I can see the traffic of these apps. This network traffic also indicates that the main application layer protocol used by these apps is still HTTP (https), yet our proxy packet capture tool is ineffective. Now, I finally encountered this obstacle in actual work, and I have to resolve it. After all, if there is something serving as a barrier to my understanding, I will feel uncomfortable all over.
2. Proxy packet capture principle
In order to understand why Fiddler or Charles is ineffective for these apps, we need to first understand the principle of proxy packet capture (of course, you don’t need to understand it, you can just watch the actual operation below to complete it. If you are interested in the principle analysis, you can come back at any time)
The packet capture software that uses a proxy, such as Fiddler or Charles, is completely different from Wireshark (Wireshark uses the network card data replication, as long as it passes through the specified network card, it will be captured). It can only be effective for application layer network protocols that use a proxy, such as the common HTTP (https) and Websocket.
Here we take HTTP as an example to briefly explain
The client needs to complete an HTTP request, usually it needs to find the server first . The client will establish a TCP connection with the target host based on the host name in the URL of the http request (actually the main name in the host will be used) and its port. After the connection is established, the client will send the http message to the target server (for more details, please refer to https://tools.ietf.org/html/rfc7232 )
Next, let me take a look at how HTTP proxy works. When we start Fiddler or Charles, we start an HTTP proxy server. This type of tool will notify the operating system, “Now I have created an HTTP proxy on the system, the IP is XXXXXX and the port is XX. If you are using Linux, you can manually notify the operating system (export http_proxy=ip:port export https_proxy=$http_proxy). If you are using a mobile device such as a mobile phone, you can tell the system that you want to use an http proxy in the current wifi settings. Now that we have told the system that we want to use a proxy, when the http client running on the system sends a request, it will no longer perform DNS resolution to connect to the target server, but will directly connect to the system to tell it the address of the proxy (the proxy’s IP and port. Note that whether it is http or https or other protocols that support proxies, they will connect to the same port). The proxy server will then establish a connection with the client, and then the proxy server will connect to the real server based on the request information.
Here is another detail. Normally, without a proxy, the request line sent by the client to the server contains only part of the URI (actually without the scheme, host name and port).
As shown in the figure above, if there is no proxy, the request line for www.baidu.com/index.html is actually GET /index.html HTTP/1.1, which is not the complete URI we often see. Because the original HTTP design did not take into account the situation of intermediate servers (i.e. proxies), the client already knows the address of the server and establishes a connection with it before sending the message, so there is no need to send the scheme, host name and port. However, this approach will have problems after the emergence of proxies. The client connects to the proxy server, but the proxy server has no way to connect to the correct server. Therefore, the request sent by the client to the proxy is actually slightly different. The client will use the complete URI in the request line so that the proxy server can resolve the address of the real server.
Now our requests are actually sent through the proxy server (Fiddler or Charles), so the proxy packet capture software not only knows all the messages of http requests and responses, but can even modify the requests and responses at any time.
3. Reasons why some applications cannot capture packets
It can be seen that the key to proxy packet capture is that the HTTP client needs to connect to the proxy server as required. Generally, we have set up the proxy at the system level. Usually, the http client is implemented as required. Before making an http request, it will check the system proxy first. If a proxy is set, the client will directly use the full uri to connect to the proxy server. Different platforms usually implement their own http clients. Although they all implement the proxy function according to the protocol requirements, they do not necessarily use the system proxy directly by default.
In reality, there are many such cases. The Flutter used in my current project is such a case. By default, Flutter will not actively use the system proxy and needs to be set separately. (Of course, I personally think that this strategy is reasonable. Although it brings inconvenience to testing and debugging, it also improves data security to a certain extent .)
Because the HTTP clients do not use the system proxy we set up, they naturally will not connect to the proxy server created by Fiddler or Charles, which ultimately prevents us from getting any requests.
4. Solution
However, now that we already know the specific reasons why Fiddler and Charles cannot capture packets, and we have mentioned the principle of proxy packet capture before, we can always find a way to solve it.
As mentioned earlier, the HTTP client used by our APP is not connected to the proxy server, which causes our proxy packet capture software to be unable to capture packets normally. Then we just need to find a way to let the client reconnect to the proxy server (of course, all this is based on the premise of not modifying the client software APP)
Method 1: Control DNS resolution and modify the DNS to make the client think that our proxy server is the target server.
Advantages: Easy to operate. By modifying the hosts of the device, it is very convenient to firstDisadvantage: You need to add a host in advance for each domain name that needs to be operated
It is difficult to modify hosts on handheld devices such as mobile phones (that is, it is difficult to implement for applications such as mobile apps)Method 2: Directly forward traffic on the network device, and forward the data sent to ports 80 and 443 on the specified terminal device directly to the target port of the proxy server
Advantages: Terminal devices connected to the network device can be configured separately, while mobile phones and other terminal devices do not need any equipment
Disadvantages: Requires separate hardware deviceMethod 3: Use VPN to forward the terminal device’s traffic to the proxy server
Advantages: No additional tests are required when using VPN software.
Disadvantages: The VPN on the terminal will directly forward all traffic by default, and it may require additional learning costs to make reasonable configurations5. Practical steps
The author directly uses the third method mentioned above (method 1 is difficult to operate for mobile APP, and method 2 may require other equipment so it is not used here). Because our test object is a mobile APP, we must first install a VPN on the mobile phone. Here we use a very convenient VPN software drony (introduction here https://github.com/SuppSandroB/sandrop/wiki/Drony-FAQ ). Drony will create a VPN on your mobile phone and redirect all traffic on the mobile phone to drony itself (not to the VPN server). In this way, drony can manage the network traffic on all mobile phones, and even configure the traffic of different APPs on the mobile phone separately.
5.1. Install drony (the phone used here is an Android device)
You can search for drony on the Internet and select the version you want to install, or download it here ( https://files.cnblogs.com/files/lulianqi/Drony_102.apk ). After the installation is complete, open the software as shown below
5.2 Open the proxy packet capture software (the proxy packet capture software used here is Fiddler)
The use of Fiddler is not introduced here. You need to open the remote proxy and install the Fiddler root certificate on your phone.
The address of the remote proxy I opened here is 192.168.2.244:8888
There are some details about certificate installation. The default root certificate of Fiddler is in cer format. Some mobile phones may only recognize pem format certificates.
Just use openssl to convert it (openssl x509 -inform der -in FiddlerRoot.cer -out FiddlerRoot.pem)
OpenSSL is usually installed in Windows/Mac. You can use the command directly. Use the generated FiddlerRoot.pem to install it on your phone (Charles uses pem certificate by default)
5.3. Configure drony forwarding
Open Drony (in OFF state), slide to the SETTING page, click Networks Wi-Fi to enter the configuration
In the network list, select the network that the current mobile phone wifi is connected to (you need to make sure that the network is connected to the Fiddler proxy server network)
Configure the proxy entry to be used for the current network (just fill in the fiddler proxy address here), and select the proxy mode as Manual.
Note that the Proxy type should be Plain http proxy
Select Direct all for Filter default value, and then click the Rule below to set the application rule.
By default, your rules should be empty. Click the plus sign above to add a rule (only messages that meet the rule requirements will be forwarded)
I would like to explain that the following operations will be demonstrated using Xianyu or Alipay. However, my current test project is not Xianyu or Alipay, nor am I an employee of these companies. I chose these two apps for demonstration because these apps are more commonly used, and the reason why they cannot capture packets is similar to that of my current project app.
Select the SSID of the current wifi in Network id
Action Select Local proxy chain
Application Select the APP that needs to be forced to proxy
If Hostname and Port are not filled in, all traffic will be forced to be proxied. Because the APP may use other network protocols, not necessarily http, and may not want to divert all traffic to the http proxy server, this configuration will be used to specify the ip and port to forward.
After completion, save it, then return to the SETTING homepage, slide to the LOG page, and click the button below to turn it on (indicates enabled)
At this time, if you start Alipay or Xianyu, we can see normal traffic on Fiddler. However, if you are as lucky as me, you may only see these Tunnel to (TLS pipeline established). If you are using Charles, you may see red crosses in the list.
Of course, the author’s Fiddler root certificate was installed successfully, and the Fiddler configuration was also correct (Chrome https packet capture on the mobile phone is normal)
Since the traffic has reached Fiddler, Drony’s work is perfectly completed. The reason why some apps cannot decrypt https messages is still a problem with our own certificates. First, let’s briefly describe the certificate verification process (if you don’t want to read these processes, you can directly see the following steps )
6. Certificate Verification Principle
Both Fiddler and Charles use a man-in-the-middle attack to replace the TLS link certificate, decrypt the message, and then encrypt it and send it to the real server.
You can refer to HTTPS Man-in-the-Middle Attack Practice (Principles and Practice) for details of Man-in-the-Middle Attack.
When there is a proxy, the client first connects to the proxy server (that is, the attacker in the figure). In fact, the client directly establishes a TLS channel with the Proxy, so the proxy of course obtains the transmission key of the TLS channel and then decrypts the message.
However, due to the existence of the certificate, the client will verify the legitimacy of the certificate and then decide whether to connect to the server. We install the root certificate in the device before using Fiddler or Charles to capture https in order to pass the client’s certificate verification.
Find any https web page in the browser and check its certificate information.
From here we can see that the certificate contains the following:
(1) Validity, also known as validity period, includes the effective time and the expiration time, which is a time interval;
(2) Subject Public Key Info, including the encryption algorithm and content of the public key;
(3) Fingerprint information. The fingerprint is used to verify the integrity of the certificate and is also the key to certificate verification. It guarantees that the certificate has not been modified. The principle is that when issuing a certificate, the publisher calculates the hash fingerprint of the entire certificate according to the fingerprint algorithm (the certificate here uses the SHA-1 and SHA-256 algorithms and has multiple fingerprints for compatibility with old clients) [the hash value of the certificate content is encrypted with the CA private key to be the fingerprint] and puts it together with the certificate. When the client opens the certificate, it also calculates the hash value of the certificate according to the fingerprint algorithm, and uses the public key of the root certificate it trusts to decrypt the hash fingerprint and calculate the original hash. If the hash values are inconsistent, it means that the certificate content has been tampered with;
(4) Certificate Signature Value and Certificate Signature Algorithm, the hash algorithm and hash value used for the certificate signature;
(5) The CA that issued the certificate: Issuer;
(6) The organization/company information Subject to which the certificate is issued;
(7) Certificate version, certificate serial number, and Extensions information, etc.
The above figure shows the process of certificate fingerprint verification. You may see that the core of the client certificate verification is actually the CA public key decrypting the original fingerprint. Where does the CA public key come from? In order to ensure security, the device system will have a list of CA public keys (root certificates) that it trusts. These CA public keys generally correspond to authoritative institutions or organizations, and when these authoritative institutions issue certificates, they will use their own private keys to sign (generate fingerprints for the certificates). This ensures that only certificates issued by authoritative institutions to various websites will be verified by the client.
Filddler does not have the private keys corresponding to the public keys in these certificates (CA only gives the private keys corresponding to the fully issued certificates to the website owner), so there is no way to complete the TLS handshake with the client. In order to complete the handshake, Filddler can only generate certificates for different sites by itself.
However, the certificate you generate must be signed with your own private key. The client cannot find the corresponding root certificate in the CA public key list it trusts, so it cannot pass the certificate verification. Therefore, Filddler requires us to install his root certificate to the device, so that the certificate issued by ourselves can pass the certificate verification and decrypt the https message by ourselves.
7. Reasons for not being able to decrypt
In fact, from the above description, it is also clear that the reason why the connection cannot be established normally to decrypt the https message is that the certificate verification failed and our root certificate installation is not complete.
Since Android 7.0, the system allows each application to define its own trusted CA set . Some applications will only trust the CA certificates pre-installed by the system by default, and will not trust the CA certificates installed by users (or the development framework used by the application will only trust the system certificates by default, because developers usually don’t care about these configurations and will not change them). In Android, the certificates installed by users are all user certificates, so whether it is Filddler or Charles, we just install their root certificates into the user certificates. These applications do not use them, so our installed certificates are invalid.
8. Solution and operation method
Now that we know the reason, there is always a way to solve it. We just need to install the root certificate of the proxy software as a system certificate.
In fact, installing the certificate to the system area is relatively simple. Just put the certificate in the specified location (/system/etc/security/cacerts/) with the specified name.
First change the name of our root certificate to .
Certificate_Hash indicates the hash value of the certificate file. Number is a suffix added to prevent the hash value of the certificate file from being the same (just use 0).
Download your own root certificate FiddlerRoot.cer, use openssl x509 -subject_hash_old -in to calculate the certificate hash, and rename the certificate to 269953fb.0 according to the hash (269953fb is the hash of the author’s certificate, and yours will definitely be different)
Then copy the 269953fb.0 file to /system/etc/security/cacerts/
After completion, we can see the certificate of the proxy software appears in the system area.
There is one more point that needs to be explained separately here. The write permission of the /system/etc/security/cacerts/ directory requires the root permission of the mobile phone.
This means that copying the certificate to this directory requires you to root your device.
Regarding rooting Android phones, usually phone manufacturers have their own official tutorials. It is recommended that you follow the official instructions to root your phone.
Take Xiaomi mobile phone as an example ( http://www.miui.com/thread-12281379-1-1.html )
Currently, the root of Xiaomi mobile phones needs to be bound to the Xiaomi account for more than 7 days before it can be unlocked. After unlocking, flash the development version to complete the root.
It should be noted that not all Xiaomi phones have corresponding development versions, so pay attention when purchasing test equipment. ( You can check whether your phone has a development version available at http://www.miui.com/download.html )
Effect
Now the certificate has also been installed in the system area. Go back to Fiddler to see the effect.
Open Xianyu again and we can see the complete https request.
Next, we find a request to modify the request return data
Use the Fiddler plug-in FreeHttp to modify the return data of this request, change the second-hand mobile phone to second-hand Ma Zong and replace the picture
(For the use of FreeHttp, please refer to Using FreeHttp to arbitrarily tamper with http messages (Usage and Implementation) )
Open Xianyu again, and you can see that the data passed through the proxy has been tampered with (note that the cache and application data of Xianyu should be cleared during the test to ensure that firstdata is requested every time the APP is opened)
9. Summary
The article discusses the challenges and solutions related to packet capture at the HTTP application layer, specifically focusing on the limitations of using proxy packet capture tools like Fiddler or Charles. It explains why these tools may be ineffective for certain applications, such as the Alipay app or apps developed with Flutter, which do not utilize systems’ default proxy settings. It also provides alternative methods for capturing packets, such as using DNS manipulation, network device traffic forwarding, or VPN traffic redirection. The article details a practical solution of using the Drony VPN on an Android device to reroute and capture mobile app traffic effectively through a proxy server. Furthermore, it examines the challenges of decrypting HTTPS messages due to certificate verification issues and offers a guide on installing a proxy’s root certificate as a system certificate to overcome these problems, ensuring complete packet visibility and control for debugging purposes.