Understanding TCP Packet Loss: Causes, Detection with Wireshark, and Mitigation Strategies

Today, I encountered a problem related to TCP packet loss, which prompted me to update the blog (20200520).

1. Packet Details

Packet Details Panel content is as follows, mainly used to analyze detailed packet information.

Frame: Physical layer, link layer

Packet: Network layer

Segment: Transport layer, application layer

1) Frame

Overview of physical layer data frames

2) Ethernet II

Ethernet frame header information at the data link layer

3) Internet Protocol Version 4

IP packet header information at the internet layer

IP Packet Header:

4) Transmission Control Protocol

Transport layer segment header information, referring here to the TCP protocol

TCP Packet Header:

5) Hypertext Transfer Protocol

Application layer information, referring here to the HTTP protocol

2. Coloring Rules

Wireshark comes with a set of default coloring rules. You can expand the frame part of the packet in the Packet Details panel to view coloring rules.

In View | Coloring Rules, open the coloring rules window. You can create, delete, select, or remove rules there.

3. Wireshark Tips

1. TCP previous segment lost (tcp previous segment not captured)
2. TCP ACKed lost segment
3. TCP window update
4. TCP duplicate ACK
5. TCP keep alive
6. TCP retransmission
7. TCP ACKed unseen segment
8. TCP port numbers reused
9. TCP retransmission
10. TCP fast retransmission
11. TCP previous segment lost (sender’s data segment lost)
12. TCP spurious retransmission

1) Packet size limited during capture

This indicates that the marked packet was not fully captured. This is generally due to the capture method used. Some operating systems by default only capture the first 96 bytes of each frame.

The full length of the fourth packet is 171 bytes, but only 96 bytes were captured.

2) TCP Previous segment not captured

If Wireshark finds that the Seq of a subsequent packet is greater than Seq+Len, it knows that a segment is missing in between.

If the missing segment cannot be found in the entire network packet (excluding disorder), it will prompt.

The Seq of packet 6 is 1449, greater than the Seq+Len of packet 5: 1+1=1, indicating that a packet of 1448 bytes in between was not captured, which is “Seq=1,Len=1448”.

‘

175: SEQ=4675 greater than 174: SEQ=Seq(4381)+0 (len)

The previous TCP segment was not captured. When the TCP connection is established, the SYN packet contains the maximum message segment length (MSS flag) of each TCP. If the sent packet is longer than the maximum message segment length, it has to be fragmented, and the fragmented packet will be marked as “TCP segment of a reassembled PDU.” These fragments share the same ACK number, but each fragment has a different Seq number.

These fragments should normally be received continuously, meaning the next seq number indicated by the previous fragment should be the seq number of the next received fragment. If the seq number of the next received fragment is not continuous compared to the previous one, Wireshark marks the fragment as TCP Previous segment not captured. As shown in the figure below, the TCP packet with an ACK number of 705 is sent in multiple fragments, one of which is a fragment with a length of 1408 that was not captured.

3) TCP ACKed unseen segment

When Wireshark finds that the ACKed packet was not captured, it will prompt.

The Seq+Len of packet 32 is 6889+1448=8337, indicating that the next packet Seq=8337.

However, we see that the Seq of packet 35 is 11233, meaning data from 8337~11232 was not captured.

4) TCP Out-of-Order

Generally due to network congestion that results in different arrival times of sequential packets, long delays, or packet loss, which requires reassembling data units because they might have arrived via different paths to your computer.

When Wireshark finds that the Seq number of a subsequent packet is less than the Seq+Len of a previous packet, it considers it out-of-order and issues a prompt.

The Seq number of packet 3362 is less than packet 3360, indicating out-of-order.

Wireshark determines TCP out-of-order based on the SEQ number in the TCP packet not being the expected next SEQ number, thus marking it as out-of-order. Therefore, when TCP out-of-order occurs, it is likely due to TCP disorder or packet loss, causing the receiver’s seq number to become non-continuous. As shown in the figure below, in the 4th packet, after the client has already received the SYN ACK from the server, the server sends another SYN ACK, which Wireshark marks as out-of-order.

As shown in the figure below, in the 7th packet, instead of receiving the TCP packet with the expected seq number 1366882, a packet with seq number 1044834 is received, indicating the packet probably arrived late, thus marked by Wireshark as out-of-order.

5) TCP Dup ACK

TCP dup ack XXX#X:This is when the receiver requests the sender to resend the sequence number XXX packet that was lost, with # indicating which attempt of the acknowledgment is being made.

When out-of-order or packet loss occurs, the receiver will receive some packets with Seq numbers larger than expected. Each time such a packet is received, it will send an acknowledgment for the expected Seq value, reflecting this to the sender.

The expected next Seq of packet 7 is 30763, but packet 8 has a Seq of 32223, indicating the packet with Seq=30763 was lost, and packet 9 was sent with Ack=30763, indicating “I need Seq=30763”.

Packets 10, 12, and 14 are also greater than 30763, so each time one of these is received, an Ack is sent.

Duplicate ack. When disorder or packet loss exists in the network, it causes the seq numbers received by the receiving end to be non-continuous. The receiving end then sends duplicate acks to the sending end, with the ack value being the expected next seq number. On receiving three or more duplicate acks, fast retransmission occurs. As shown below, in packet 315, the client sends an ack=126951 feedback to the server, expecting the next received packet to have seq=126951. But the next packet received has seq=128211, so due to the non-continuity of the seq, Wireshark marks it as TCP Previous segment not captured. In packet 317, the client sends another duplicate ack=126951 packet, marking it as the first retransmission with a # after it showing 1. In packet 318, the TCP packet with seq=126951 is received. In packet 319, all TCP packets up to seq=129471 are fully received, so the client sends ack=129471 feedback.

6) TCP Fast Retransmission

When the sender receives 3 or more [TCP Dup ACK], it becomes aware that the previously sent packet may have been lost, so it retransmits it quickly.

TCP fast retransmission. The TCP protocol sets up a fast retransmission mechanism to avoid a slow start from overly impacting the transmission rate. Fast retransmission works by being triggered upon receiving 3 or more duplicate ack feedbacks. Fast retransmission does not need to wait for RTO timeout. As shown below, in packet 325, the client sends ack=133251 feedback to the server, indicating the next expected packet to be received is seq=133251 from the server; in packet 326, the server sends a packet with seq=135771 to the client, which does not align with the client’s expectation, hence the client in packet 327 retransmits ack=133251, restating the expectation of receiving seq=133251. Wireshark marks the duplicate ack as TCP Dup ACK, indicating the frequency of retransmission with a # following. In packet 328, the server sends a packet with seq=137031 to the client again, which still does not meet with the client’s expectations, so the client retransmits ack=133251 in packet 329 again. In packet 330, having received 3 duplicate acks, the server triggers fast retransmission and retransmits the TCP segment with seq=133251.

7) TCP Retransmission

If a packet is actually lost, and there are no subsequent packets that can trigger [Dup Ack] at the receiving end, it won’t retransmit quickly.

In this case, the sender has to wait for a timeout before retransmitting.

Packet 1053 was sent without receiving a corresponding Ack, so it could only be retransmitted after over 100 milliseconds had passed.

TCP retransmission. When two instances of the same packet data are captured, Wireshark determines that retransmission has occurred, and Wireshark did not capture an ack for the initial transmission of the packet. Hence, Wireshark considers the retransmission valid and marks it as TCP Retransmission. Based on software’s judgment mechanism, if the capture point is on the client, TCP retransmissions are generally uplink packets. This is because the client did not receive a feedback ack from the server at that time, unsure if the server received the initial packet. After the RTO timeout, client retransmission is triggered. There are two situations: 1) the server received the initial packet, but the feedback ack was lost, and the client did not receive it; 2) the server did not receive the initial packet, so no feedback ack was issued. For the first situation, if the capture point were on the server, Wireshark might mark the retransmitted packet from the client as a TCP Spurious Retransmission. As shown below, the red-line TCP packet is a retransmission packet, and Wireshark adds a retransmission reason to the packet, due to TRO timeout leading to retransmission, and pinpoints the sequence number 45 of the initial transmission and the current RTO timeout period.

8) TCP zero window

The “win” in the packet represents the size of the receiving window, and when Wireshark finds “win=0” in a packet, it issues a prompt.

TCP sliding window is 0. When the packaging rate from the sender exceeds the receiving rate of the receiver, TCP window size on the receiver keeps getting smaller. When the window size on feedback ack from the receiver is 0, Wireshark marks it as TCP Zero window. At this moment, the sender will pause sending data until it receives a sign from the receiver that the window size!=0.

9) TCP window Full

This prompt indicates that the sender of the packet has exhausted the receiving window declared by the other party.

When Wireshark calculates that the Middle East has 65535 bytes not acknowledged, it will issue this prompt.

[TCP window Full] indicates the sender temporarily cannot send data;

[TCP zero window] indicates sending temporarily cannot receive data.

TCP window full. It refers to the data sent by the sender reaching the upper limit of the receiving window. The sender pauses transmission and waits for a new notification of the receiving window. As shown below, in the ack feedback from the client to the server, the next expected sequence is seq=288961, but the receiver’s window is only 960, and the server, after receiving ack, sends 960 bytes of data, TCP window full. Note that len=1004 is the length of the entire IP packet, and 44 bytes must be subtracted for the IP header, resulting in 960 bytes.

10) TCP segment of a reassembled PDU

Wireshark can virtually aggregate TCP packets, which belong to the same Application layer PDU.

After receiving a large block of data from the upper layer, the TCP layer disassembles it into segments before sending it out. When a host responds to a query or command with a large amount of data (information) that exceeds TCP’s maximum MSS,

the host will send multiple packets to transmit this data (note: these packets are not being fragmented).

11) Time-to-live exceeded (Fragment reassembly time exceeded)

Indicates that the sender of this packet had received some fragments previously but, due to some reason, could not assemble them.

12. TCP window update

TCP window update. When there is an abrupt change in the TCP window of the receiver, the receiver, through a TCP window update message, informs the other party of the current receiving window size. As shown below, the TCP Window Update also carries feedback ack, and the ack sequence number is the same as the previous feedback ack sequence number. Nevertheless, it’s not a duplicate ack.

Line 109:

The server sends a TCP window update to the client, indicating the buffer is cleared. The server now has an ample window size of 43320.

TCP window update is a state in TCP communication that can occur for various reasons, but ultimately it boils down to the sender transmitting data faster than the receiver reads it, causing the receiver to release part of the buffer space to accommodate incoming data. Afterward, a Windows Update is sent to the sender, indicating the rate at which data should be sent, ensuring transmission and reception return to normal.

13. TCP acked unseen segment

A feedback ACK pointing to an unseen TCP segment. This means the feedback ACK is for a TCP packet not available in Wireshark, possibly because Wireshark missed capturing the packet but captured the ack for that packet from the opposing side. As depicted below, the packet marked as ack unseen segment carries ack=2721, which seems to acknowledge the packet with seq=1361. However, this ack also acknowledges the packet with seq=1, not captured, hence prompting Wireshark to mark the feedback ack as an ack unseen segment. The subsequent graphics demonstrate that because the remote already provided ack=2721 feedback, the sending end’s packet with seq=1 was indeed received, although Wireshark might have simply missed capturing it.

14. TCP RST

TCP Reset. This is a way to terminate an abnormal TCP connection set by the TCP protocol through a flag under a reset=1 marker. When a TCP connection cannot end via the normal four-way handshake, a party may terminate the TCP connection by sending a TCP packet marked with a reset flag. As shown below, the sender sends data over two TCP streams; the screenshot shows the receiver first provides a TCP window=0 feedback to the sender, advising pause in sending data, and subsequently sends a TCP RST label, releasing the TCP connection. It might be inferred that the receiving program crashed suddenly, causing data in the buffer to not be cleared, and eventually triggering the receiver system to send a TCP reset, ending the TCP connection.

Problem

The problem we encountered: The client stuttered seriously while downloading and playing audio simultaneously under a weak network condition:

Reasons for stuttering: 1. The default MTU for HTTP is 1400, and each TCP package size is 1400, but TCP cannot ensure data arrives in order. The client completes receiving the entire data before sorting data packets, subsequently passing ordered data to the application layer HTTP. 2. TCP packets may undergo retransmission under poor network conditions. 3. The client receives and plays simultaneously, i.e., playing according to packet seq. If a packet undergoes TCP retransmission, the client must wait to receive this packet before continuing playback. Therefore, stuttering can occur. 4. We did not see the seq=29401 packet in Wireshark, not necessarily because the server did not issue it. Wireshark prompt: TCP Previous segment not captured. Wireshark might not have captured this packet. Solution: Currently, receiving and playing simultaneously has a high probability of TCP retransmission under weak network conditions, leading to stuttering. Interruption and resume downloading play significantly reduce stuttering.

1. Packet Loss:

Wireshark capture first notices: TCP Previous segment not captured, indicating packet data loss:

501: seq=28001, ACK=140

502: seq=30801, ACK=140 (TCP Previous segment not captured)

i.e., packet seq=29401, ACK=140 was lost.

503: seq=40, ACK=29401 requests the server resend that packet,

504: seq=32201, ACK=140 Packet arrived,

Packets 505, 507, 509 (TCP Dup ACK) consistently telling the server the packet was lost,

2. Retransmission of Data Packet: Results only after 547, the server retransmitted TCP packets.

The delay in waiting would certainly cause audio playback stutter.

3. Continue to Request Retransmission: seq=30801, ACK=140 (550, 552, 554…TCP Dup ACK).

In fact, this packet was sent before, but the client did not cache this data.

4. Re-retransmission of Data Packet: Until 573, the server retransmitted TCP packets.

Client audio playback stuttered quite severely.