Comprehensive Guide to TCP/IP Protocol: Structure, Security Risks, and Prevention Methods

TCP/IP protocol, as the most popular Internet protocol today, was not designed with future security needs in mind, resulting in numerous security issues within the protocol. The security flaws in the protocol, combined with the existence of computer viruses, pose significant risks to the network environment. This article analyzes TCP/IP and its security issues, introduces the current security technologies from both protocol and application perspectives, and finally proposes a security prevention system architecture.

  1. Introduction to TCP/IP Protocol

  The TCP/IP protocol originated from a packet-switched network research project funded by the U.S. government in the late 1960s and had developed into the most common form of networking between computers by the 1990s. It is a truly open system that supports the interconnection of hosts with different operating systems and different types of networks.

  The TCP/IP protocol is a combination of multiple protocols at different levels, usually divided into four layers.

  (1) Link Layer, sometimes referred to as the Data Link Layer or Network Interface Layer, is responsible for handling the physical interface details with the cable, such as receiving IP packets from the network layer, adding a physical header, and sending them onto the line.

  (2) Network Layer, which receives packets from the transport layer, adds an IP header, and is responsible for packet fragmentation and reassembly, as well as routing during transmission.

  (3) Transport Layer, which primarily provides end-to-end communication for applications on two hosts. It mainly includes two protocols: TCP (Transmission Control Protocol) and UDP (User Datagram Protocol), where TCP provides reliable connection-oriented service, and UDP provides unreliable connectionless service.

  (4) Application Layer, which handles the details of specific applications.

  The IP protocol is the only core protocol at the TCP/IP network layer. This is a key reason for the widespread application of the TCP/IP protocol. Regardless of the network protocol used, all TCP, UDP, and other protocol packets are directly encapsulated by IP datagrams and transmitted as their data fields, providing great flexibility for interconnecting heterogeneous networks. The IP protocol is mainly responsible for packet fragmentation and reassembly, as well as routing.

  2. Security Risks of TCP/IP Protocol

  One significant cause of operating system vulnerabilities is the attack points brought about by the protocol’s own flaws. Network protocols are rules that computers follow to interconnect. The mainstream protocol used in current interconnected networks, TCP/IP, was designed with an emphasis on openness and convenience without careful consideration of security, resulting in serious security vulnerabilities in many network protocols, leaving many security risks for the Internet. Additionally, some security vulnerabilities caused by network protocol flaws can be directly exploited by hackers to attack victim systems. This article discusses the security issues inherent in the TCP/IP protocol and its daemons in detail, pointing out attacks targeting these security risks.

  1. Security Issues of TCP Protocol

  TCP uses a three-way handshake to establish a connection. The first packet in the handshake is a SYN packet; the second packet is a SYN/ACK packet, indicating it acknowledges the first SYN packet and continues the handshake process; the third packet is merely an acknowledgment, represented as an ACK packet. If A is the initiator and B is the responder, the potential threats include:

  1. An attacker eavesdropping on the SYN/ACK packet sent by B.

  2. An attacker sending an RST packet to B, followed by a SYN packet, impersonating A to initiate a new connection.

  3. B responds to the new connection and sends a SYN/ACK packet.

  4. The attacker then impersonates A to send an ACK packet to B.

  This way, the attacker achieves the goal of disrupting the connection, and if the attacker inserts harmful data packets, the consequences can be even more severe.

  TCP treats the data transmitted through the connection as a byte stream, numbering the transmitted bytes with a 32-bit integer. The initial sequence number (ISN) is generated during the TCP handshake, and its generation mechanism is related to the protocol implementation. An attacker only needs to send a connection request to the target host to obtain the ISN of the last connection, then measure the round-trip time (RTT) of the transmission path multiple times to get the RTT between the attacking host and the target host. Knowing the ISN of the last connection and the RTT, it is easy to predict the ISN of the next connection. If an attacker impersonates a trusted host to send a TCP connection to the target host and predicts the TCP sequence number of the target host, the attacker can forge harmful data packets to be accepted by the target host.

  Security Issues of IP Protocol

  The IP protocol provides connectionless packet transmission between interconnected networks. The IP protocol sends IP packets based on the destination address field in the IP header. In other words, when routing IP packets, the IP protocol does not check the source address provided in the IP header and assumes that the source address in the IP header is the IP address of the machine sending the packet. This creates problems and can lead to illegal intrusions for many services that rely on IP source address verification. The most important of these is various attacks caused by IP spoofing.

  For example, some network firewalls only allow trusted IP packets to pass through. However, because the IP address does not verify whether the IP source address in the IP packet is the true address of the source host, attackers can use IP source address spoofing to bypass such firewalls. Additionally, some network applications that use IP addresses as the basis for security permissions can be easily exploited by attackers using IP source address spoofing to gain privileges, causing serious damage to the attacked party. In fact, any attacker can exploit the characteristic of IP not verifying the source address in the IP header by filling in a forged IP address to attack, making it difficult to be detected.

  3. Prevention of TCP/IP Protocol Security Issues

  Prevention of TCP Protocol Security Issues

  For SYN Flood attacks, there is currently no completely effective method, but the following measures can help prevent them:

  1. Set appropriate kernel parameters for the system to force the reset of timed-out SYN request connection packets, while shortening the timeout constant and lengthening the waiting queue to quickly handle invalid SYN request packets.

  2. It is recommended to make some configuration adjustments on the router in the network segment, including limiting the flow and number of SYN half-open packets.

  3. It is recommended to implement necessary TCP interception at the front end of the router so that only packets that have completed the TCP three-way handshake process can enter the network segment, effectively protecting the servers within the network segment from such attacks.

  Prevention of IP Protocol Security Issues

  1. Abandon address-based trust strategies. This is the simplest method.

  2. Implement packet filtering. If the network is connected to the Internet through a router, the router can be used for packet filtering. Ensure that only the internal LAN can use trust relationships, and handle hosts on the internal LAN with caution when dealing with external hosts. The router can filter out all requests from external sources attempting to establish connections with the internal network.

  3. Use encryption technology. A simple method to prevent IP spoofing is to require encrypted transmission and verification during communication. When multiple methods coexist, encryption is likely the most suitable.

  4. Security of Each Layer of TCP/IP and Methods to Enhance Security

  1. Network Layer Security

  Over the past decade, some proposals have been made to standardize network layer security protocols. For example, Security Protocol 3 (SP3) was developed by the U.S. National Security Agency and the National Institute of Standards and Technology as part of the Secure Data Network System (SDNS). The Network Layer Security Protocol (NLSP) was developed by the International Organization for Standardization as a security protocol standard for the Connectionless Network Protocol (CLNP). Integrated NLSP (I-NLSP) was proposed by the U.S. National Institute of Technology and Standards to include a unified security mechanism for both IP and CLNP. SWIPE is another network layer security protocol proposed and prototyped by Ioannidis and Blaze. All these proposals share more similarities than differences. Essentially, they use IP encapsulation technology. In essence, plaintext packets are encrypted and encapsulated in an outer IP header for routing on the Internet. Upon reaching the other end, the outer IP header is stripped off, the message is decrypted, and then delivered to the recipient.

  The main advantage of network security is its transparency, meaning that providing security services does not require any changes to applications, other communication layers, or network components. Its main disadvantage is that the network layer generally does not distinguish between packets belonging to different processes and corresponding policies. All packets going to the same address are processed using the same encryption keys and access control policies, which may lead to the inability to provide the required functionality and performance degradation. To address these issues with host-oriented key distribution, RFC 1825 allows (and even recommends) user-oriented key distribution, where different connections receive different encryption keys. However, user-oriented key distribution requires significant changes to the operating system kernel.

  In short, the network layer is very suitable for providing host-to-host security services. Corresponding security protocols can be used to establish secure IP tunnels and virtual private networks on the Internet. For example, using its IP packet encryption and decryption functions can easily enhance the defense capabilities of firewall systems. RSA Data Security has initiated an initiative to promote the joint development of virtual private networks by multiple firewall and TCP/IP software vendors, known as the S-WAN (Secure Wide Area Network) initiative, aiming to establish and recommend network layer security protocol standards.

  2. Transport Layer Security

  In network application programming, the generalized inter-process communication (IPC) mechanism is usually used to interact with security protocols at different levels. The first idea to provide security services on the Internet is to enhance its IPC interface, such as BSD Sockets, by including mutual authentication of both entities and the exchange of data encryption keys. Netscape Communications followed this approach and developed the Secure Sockets Layer (SSL) protocol, which is built on reliable transport services (such as those provided by TCP/IP).

  The main advantage of network security mechanisms is their transparency, meaning that providing security services does not require any changes to the application layer. This is not achievable for the transport layer. In principle, any TCP/IP application that uses transport layer security protocols, such as SSL or IPC, must undergo some modifications to add corresponding functionality and use different IPC interfaces. Therefore, the main disadvantage of transport layer security mechanisms is that modifications are required for both the transport layer IPC interface and the application programs. However, compared to security mechanisms at the Internet layer and application layer, the modifications here are relatively small. Another disadvantage is that it is challenging to establish security mechanisms at the transport layer for UDP-based communication. Compared to network layer security mechanisms, the main advantage of transport layer security mechanisms is that they provide process-to-process (rather than host-to-host) security services. This achievement, combined with application-level security services, can take security a step further.

  3. Application Layer Security

  Network layer security protocols allow adding security attributes to data channels between hosts (processes), meaning that the actual data channel is still established between hosts (or processes), but it is not possible to distinguish the security requirements of a specific file transmitted on the same channel. For example, if a secure IP channel is established between one host and another, all IP packets transmitted on this channel will be automatically encrypted. Similarly, if a secure data channel is established between two processes using a transport layer security protocol, all messages transmitted between the two processes will be automatically encrypted.

  Generally speaking,