To troubleshoot network issues, it’s essential to understand the TCP/IP principles, as the truth lies in the details of each data packet. Below are some critical parameters related to retransmissions.
Other references:
/proc/sys/net/ipv4/* Variables: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
When a request packet is sent out, a timer is initiated. If the timer expires without receiving an ACK, the request is retransmitted. This continues until the maximum number of retransmissions is reached or an ACK is received.
If the receiver finds the packet sequence number to be incorrect, it will resend the ACK for the missing packet. If the sender receives three duplicate ACKs for the same sequence number, it will activate the fast retransmission mechanism to resend the packet associated with that ACK. For specific details, refer to the following:
1 Network device port or optical module anomalies may cause checksum failures.
2 Network routing convergence fluctuations
3 Bugs in host network drivers or network devices, etc.
Using tsar -tcp -C allows monitoring of the tcp retransmission attribute, which refers to the number of retransmissions.
tsar –tcp -C | sed ‘s/:/_/g;s/=/ /g’ | xargs -n 2
Interested individuals can execute the following monitoring script directly to obtain TCP-related status monitoring data applicable to open-falcon.
1 Capture packets on machines experiencing packet loss retransmissions and analyze them with Wireshark. As retransmissions do not occur constantly, the capture command must be executed continuously to capture retransmitted packets. Use Wireshark to open the tcpdump results and enter the following in the search box to get the result shown below:
Figure 1 shows that the server performed three retransmission actions.
2 Given the high volume of packets, we can use Wireshark to capture the TCP stream related to retransmissions.
Figure 2 Track Stream -> TCP Stream can obtain the packets related to retransmissions.
Figure 3 shows the request and response between the client and server.
3 Analyze retransmissions
It is particularly important to note
NO 67, 68 The client did not receive the correct packet data for some reason and sent dup ack to the server, as referenced in the basic knowledge mentioned.
The time difference between NO.68 and NO.69 is 200ms (all others are less than 1ms), the server waits for a timeout and then retransmits.
NO 73-74 The client sent a fin packet and actively closed the connection.
This case occurred only once and wasn’t reproduced. Through capturing and analyzing packets, no clear conclusion was reached.
This article summarizes the troubleshooting process for TCP retransmission issues encountered in my work, focusing on the general problem-solving approach and specific practices, with less emphasis on theoretical knowledge. Those interested can read the recommended articles to gain a deeper understanding of TCP’s working mechanism.
TCP retransmission series articles
https://www.cnblogs.com/lshs/p/6038516.html https://www.cnblogs.com/lshs/p/6038527.html https://www.cnblogs.com/lshs/p/6038536.html
Network performance troubleshooting with TCP retransmissions and duplicate ACKs: https://www.kancloud.cn/digest/wireshark/62473
Learn Wireshark in a comprehensive way: https://www.kancloud.cn/digest/wireshark