Top 5 TCP/IP Network Security Issues and Solutions for Enterprise Network Managers

In this article, the author focuses on analyzing the five major network security issues faced by the TCP/IP protocol stack and introduces the countermeasures that enterprise network security managers can take when facing these problems.

1. IP Spoofing

IP Spoofing can be understood as a technique where one host device impersonates another host’s IP address to communicate with other devices for some purpose. As early as 1985, a Bell Labs engineer, Robbert Morris, proposed the concept of IP Spoofing in his article “A weakness in the 4.2bsd UNIX TCP/IP software.” Interested readers can refer to the original text: http://www.coast.net/~emv/tubed/archives/Morris_weakness_in_TCPIP.txt.

However, it should be noted that relying solely on IP Spoofing technology cannot complete a full attack effectively, as the existing IP Spoofing technology is a “blind” intrusion method.

Generally speaking, an IP spoofing attack has six steps:

(1) First, temporarily paralyze the trusted host’s network to avoid interference with the attack;

(2) Then connect to a port on the target machine to guess the ISN base value and increment pattern;

(3) Next, disguise the source address as the trusted host and send a SYN flag data segment to request a connection;

(4) Then wait for the target machine to send a SYN+ACK packet to the paralyzed host;

(5) Finally, disguise as the trusted host again and send an ACK to the target machine, with the data segment carrying the predicted ISN+1 of the target machine;

(6) The connection is established, and command requests can be sent.

Here are its two key steps:

(1) Disable the trusted host

To impersonate the trusted host without being exposed, it needs to be completely disabled. Since the attacker will replace the real trusted host, they must ensure that the real trusted host cannot receive any valid network data, or they will be exposed. There are many ways to achieve this (such as SYN flood attacks, Land attacks, etc.).

(2) Sequence number sampling and guessing

To attack the target host, the attacker must know the sequence number of the target host’s data packets. How is this usually predicted? Typically, a normal connection is first established with a port (such as port 25) of the target host. This process is often repeated N times, and the last ISN sent by the target host is stored. Then, the round-trip time between the attacker’s host and the trusted host needs to be estimated, which is calculated by averaging multiple statistics. If the round-trip connection increases by 64,000, it can be estimated that the ISN size is 128,000 times half the round-trip time. If the target host has just established a connection, add 64,000. Once the ISN size is estimated, the attack begins. If the estimated sequence number is accurate, the fake TCP data packet will be placed in the target host’s buffer.

However, in actual attacks, it is often not so lucky. If the estimated sequence number is less than the correct value, it will be discarded. If the estimated sequence number is greater than the correct value and within the buffer size, the data will be considered future data, and the TCP module will wait for the missing data. If the estimated sequence number is greater than the expected number and not within the buffer size, TCP will discard it and return an expected data sequence number. After impersonating the trusted host’s IP, the host remains paralyzed, and a connection request is sent to the target host’s attacked port (such as port 25). The target host immediately responds to the connection request, sending an updated SYN+ACK confirmation packet to the trusted host. Since the trusted host is still paralyzed, it cannot receive the packet. The attacker then sends an ACK data packet to the target host, using the previously estimated sequence number plus 1. If the attacker estimates correctly, the target host will accept the ACK, and the connection will be officially established, allowing data transmission to begin.

For external network spoofing, the prevention method is simple: just add a restriction setting on the external router of the local area network, prohibiting packets claiming to come from within the network.

For preventing IP spoofing attacks from outside the local area network, a firewall can be used. However, for internal attacks, setting up a firewall is ineffective. In this case, attention should be paid to whether the internal network router supports internal interfaces. If the router supports two interfaces for the internal network subnet, caution is needed as it is susceptible to IP spoofing.

Monitoring packets to check for IP spoofing attacks is very effective. Using packet inspection tools like netlog to verify the source and destination addresses of the information, if packets are found to come from more than two different addresses, it indicates that the system may have been subjected to an IP spoofing attack, and a hacker outside the firewall is attempting to invade the system.

2. SYN Flooding

SYN Flooding is one of the most effective and popular forms of DoS attacks. It exploits the flaw in the TCP three-way handshake protocol by sending a large number of SYN connection requests with forged source addresses to the target host, consuming the target host’s resources, and rendering it unable to provide services to legitimate users.

In the initial phase of a TCP session, there is a so-called “three-way handshake” process: tracking the amount of data sent and received to synchronize the sending and receiving of data segments, determining the data acknowledgment number based on the amount of data received, and when to terminate the connection after data transmission and reception are complete, establishing a virtual connection. To provide reliable transmission, TCP numbers the data packets in a specific order before sending new data and waits for acknowledgment messages after these packets are delivered to the target machine. TCP is always used to send large amounts of data. When an application needs to acknowledge received data, it also uses TCP. Since TCP needs to keep track of everything, it incurs additional overhead, making the TCP format somewhat complex.

The steps of the TCP three-way handshake are as follows:

(1) Suppose host A wants to communicate with host B and establish a TCP connection. First, host B (the server in this case) must run a server process and issue a “passive open” command to TCP. The server process then continuously probes the port to see if any client process has a connection request and remains in a “listening” state. The client host A’s application process sends an “active open” command to its TCP, specifying the IP address and port number to establish a TCP connection. Host A’s TCP then sends a connection request packet to host B’s TCP. The TCP packet specifies the IP address (implicitly specified in the TP datagram) and port number to connect to, sets the maximum acceptable TCP segment size, and includes some user data, with SYN=1 and ACK=0. This is called the “first handshake.”

(2) When host A’s connection request reaches host B, host B’s TCP checks if there is a process listening on the port. If not, it sends a response with RST=1, rejecting the connection; otherwise, it leaves the TCP segment for the “listening” process. The “listening” process sends back a response TCP segment with SYN=1, ACK=1, acknowledgment number ACKSEQ=X+1, and selects a sequence number SEQ=Y. This is the “second handshake.”

(3) After host A receives the acknowledgment packet from host B, it sends an acknowledgment TCP segment to host B with SYN=1, ACK=1, SEQ=X+1, and ACKSEQ=Y+1, completing the “third handshake.”

In a SYN Flooding attack, the hacker machine sends a large number of TCP SYN packets with forged source addresses to the victim host. The victim host allocates necessary resources and returns a SYN+ACK packet to the source address, waiting for the source to return an ACK packet, as shown in Figure 1. Since the source address is forged, the source never returns an ACK packet. The victim host continues to send SYN+ACK packets and places the half-open connection in the port’s backlog queue. Although most hosts have timeout mechanisms and default retransmission counts, the length of the half-open connection queue is limited. If a large number of TCP SYN packets are continuously sent to the victim host, the half-open connection queue will quickly fill up, the server will refuse new connections, and the port will be unable to respond to connection requests from other machines, eventually exhausting the victim host’s resources.

TCP/IP network security

Figure 1 SYN Flooding attack diagram

Currently, there are two relatively effective technologies for defending against SYN Flooding attacks.

(1) SYN-cookie technology

Generally, when a server receives a TCP SYN packet, it immediately allocates a buffer for the connection request and returns a SYN+ACK packet, forming a half-open connection. SYN Flooding exploits this by sending a large number of SYN connection requests with forged source addresses without completing the connection, consuming significant server resources.

SYN-cookie technology addresses this resource allocation flaw in the standard TCP connection establishment process by changing the resource allocation strategy. When the server receives a SYN packet, it does not immediately allocate a buffer but generates a cookie using the connection information and uses this cookie as the initial sequence number of the SYN+ACK packet to be returned. When the client returns an ACK packet, the server calculates the cookie based on the packet header information and compares it with the first 24 bits of the acknowledgment sequence number (initial sequence number + 1). If they match, it is a legitimate connection, and resources are allocated to establish the connection.

The cleverness of this technology lies in avoiding resource allocation before the connection information is fully received, rendering the resource consumption of SYN Flooding attacks ineffective. The key to implementation is the calculation of the cookie. The cookie calculation should include the state information of the current connection, preventing attackers from forging the cookie. The cookie calculation process is as follows.

① When the server receives a SYN packet, it calculates a message digest mac:

mac = MAC(A, k)

MAC is a message authentication code function in cryptography, a keyed hash function that provides the necessary security for cookie calculation. A is a concatenation of the client and server IP addresses and port numbers, and the parameter t: A = SOURCE_IP || SOURCE_PORT || DST_IP || DST_PORT || t; K is a server-specific key; the time parameter t is a 32-bit time counter that increments every 64 seconds;

② Generate the cookie:

cookie = mac(0:24): representing the 0th to 24th bits of the mac value;

③ Set the initial sequence number of the SYN+ACK packet to be returned as follows:

· The high 24 bits are replaced by the cookie;

· The next 3 bits are replaced by the maximum segment size (MMS) requested by the client;

· The last 5 bits are t mod 32.

When the client receives the SYN+ACK packet from the server, it returns an ACK packet with the cookie (the acknowledgment number is the initial sequence number of the SYN+ACK packet sent by the server plus 1, so it does not affect the high 24 bits). The server recalculates the cookie and compares it with the first 24 bits of the acknowledgment number. If they match, it indicates that the connection has not been tampered with and is legitimate, and the server completes the connection establishment process.

SYN-cookie technology effectively defends against SYN Flooding attacks by implementing a stat

③ Upon intercepting a SYN ACK packet from the server (with the destination address as addr), it indicates that the server has established a half-open connection for the connection request from addr. To prevent an excessive number of half-open connections, send an ACK packet to the server to establish the connection. Start a timer; if it times out without receiving an ACK packet, it indicates that addr is unreachable. If addr’s status is GOOD, change it to NEW; if it’s NEW, change it to BAD; if it’s BAD, it remains unchanged.

The address status transition diagram is shown in Figure 2.

TCP/IP network security

Figure 2 Address Status Transition Diagram

Let’s analyze how the address status monitoring method can defend against SYN Flooding attacks.

For a SYN packet with a spoofed source address, if the source address appears for the first time, its status is NEW. When a SYN+ACK packet from the server is intercepted, it indicates that the server has established a half-open connection for the source address. At this point, the monitoring program sends an ACK packet on behalf of the source address to complete the connection, ensuring the half-open connection queue is not overly populated. The timer starts; since the source address is spoofed, no ACK packet will be received. After timeout, the monitoring program sends an RST packet, and the server releases the connection, changing the source address status to BAD. Subsequently, for each SYN packet from this source address, the monitoring program will proactively send an RST packet.

For a legitimate SYN packet, if the source address appears for the first time, its status is NEW. The server responds to the request by sending a SYN+ACK packet, and the monitoring program sends an ACK packet, completing the connection. Subsequently, an ACK from the client will quickly arrive, changing the source address status to GOOD. The server can efficiently handle duplicate ACK packets.

3. ACK Flooding

ACK Flooding attacks occur after a TCP connection is established, where all transmitted TCP packets have the ACK flag set. When a host receives a packet with the ACK flag, it must check whether the connection tuple exists. If it does, it checks the packet’s state legality before passing it to the application layer. If the packet is found illegal, such as pointing to a non-open port, the host’s OS protocol stack responds with an RST packet to indicate the port’s non-existence.

Here, the server performs two actions: table lookup and ACK/RST response. This attack method is evidently less impactful than SYN Flooding, so attackers must use high-volume ACK small packets to affect the server. Based on our understanding of the TCP protocol, random source IP ACK small packets should be quickly discarded by the server, as the server’s TCP stack lacks state information for these ACK packets. However, testing reveals that some TCP services are sensitive to ACK Flooding, such as JSP Servers, which struggle to handle normal connection requests under a modest ACK packet attack. For Apache or IIS, 10kbps of ACK Flooding is not a threat, but higher volumes can cause excessive network card interrupts and overload, leading to unresponsiveness. ACK Flooding can harm routers and network devices and significantly impact server applications.

If no port is open, the server will directly discard packets, consuming CPU resources. If the port is open, the server responds with an RST.

Symmetry judgment can be used to analyze the presence of an attack. Symmetry judgment means receiving packets significantly exceeds sending packets, as attackers typically use numerous ACK packets and send small, consistent packets to increase attack speed. This can indicate ACK Flooding, but it is often mixed with other attack methods, leading to misjudgment.

Some firewall responses include creating a hash table to store TCP connection “states,” simplifying state checks compared to the host’s TCP stack implementation. For example, not checking sequence numbers or handling packet reordering, but merely counting if ACK packets pass through the “connection” (i.e., tuple) within a certain time, to “roughly” determine if the “connection” is “active.”

4. UDP Flooding

UDP Flooding is an increasingly rampant traffic-based DoS attack, with a simple principle. Common scenarios involve using numerous small UDP packets to overwhelm DNS servers, Radius authentication servers, or streaming media servers. 100kbps of UDP Flooding can often cripple backbone devices like firewalls, causing network segment paralysis. Since UDP is a connectionless protocol, attackers can send many small UDP packets with spoofed source IP addresses. However, because UDP is connectionless, opening a UDP port for service allows targeted attacks on that service.

In normal applications, UDP packet bidirectional traffic is generally equal, with random and varied sizes and content. In UDP Flooding, a large number of UDP packets with fixed content and size appear on one side targeting the same destination IP.

Unlike TCP, the UDP protocol is connectionless, and UDP application protocols vary greatly, making UDP Flooding defense challenging. Protection must be tailored to specific circumstances.

· Packet size judgment: For large packet attacks, use methods to prevent UDP fragmentation. Set packet reassembly size based on attack packet size, usually no less than 1500. In extreme cases, consider discarding all UDP fragments.

· Attack port as service port: Set detection of UDP maximum packet length based on the service’s maximum UDP packet length to filter abnormal traffic.

· Attack port as non-service port: One option is to discard all UDP packets, possibly affecting normal services; another is to establish UDP connection rules, requiring all UDP packets to first establish a TCP connection to the port. This requires advanced firewalls or other protective equipment support.

Using firewalls at critical network points to filter harmful data from unknown sources can effectively mitigate UDP Flooding attacks. Additionally, users should take the following measures within their networks.

· Disable or filter monitoring and response services.

· Disable or filter other UDP services.

· If users must provide external access to some UDP services, use proxy mechanisms to protect those services from abuse.

· Monitor the network to understand which systems use these services and watch for abuse signs.

· For small servers, directly use firewall rules to block them.

5. Connection Flooding

Connection Flooding is a typical and highly effective attack method using low traffic to impact high-bandwidth network services. This attack method is becoming increasingly rampant. The principle is to use real IP addresses to initiate numerous connections to the server and maintain them for extended periods without releasing them, consuming server resources and causing excessive residual connections (WAIT state), reducing efficiency, and potentially exhausting resources, preventing responses to other client connection requests.

One attack method is to initiate numerous connection requests per second, similar to fixed source IP SYN Flooding attacks, but using real source IP addresses. Firewalls can limit the number of connections per source IP per second for protection. However, tools now use slow connections, establishing a connection with the server every few seconds, and after successful connection, not releasing it while periodically sending junk packets to maintain the connection. This allows a single IP address to establish hundreds or thousands of connections with the server, which has a limited connection capacity, achieving a denial-of-service effect.

Additionally, during large-scale worm outbreaks, due to the simplicity of worm code, many packets with the same source IP address appear during propagation, exhibiting widespread scanning behavior for TCP worms. This is important when identifying Connection Flooding.

The general manifestation of this attack is using the netstat –an command on the attacked server to observe numerous connection states from a few source addresses. Statistics may show abnormal connection numbers compared to usual, and after reaching a certain threshold, fluctuations indicate approaching performance limits. Thus, the judgment principle for this attack is: traffic may not be significant, possibly very small; numerous ESTABLISH states appear; the total number of new ESTABLISH states fluctuates.

Preventive measures for this attack include:

· Actively clear residual connections.

· Block malicious connection IPs.

· Limit the number of connections per source IP.

· Protect specific URLs.

· Reverse lookup the source behind a proxy initiating HTTP Get Flood.