Understanding Malicious Traffic IPs: Effective Blocking and Detection Techniques

TCP reassembly is resource-intensive, so it’s important to find some optimization algorithms that avoid TCP reassembly as much as possible while not compromising the integrity of the traffic. Four current considerations include configuration constraints, lists categorized as red, yellow, and green, sampling algorithms, and a reasonable aging duration.

Configuration Constraints

Lists Categorized Red, Yellow, and Green

Red list: Malicious traffic IPs, all traffic associated with the current IP gets blocked immediately without any detection.

Yellow list: Malicious traffic IPs, all traffic associated with the current IP skips TCP reassembly but undergoes single packet inspection.

Green list: Legitimate traffic IPs, all traffic associated with the current IP skips TCP reassembly but undergoes single packet inspection.

1. Upon detecting malicious traffic, the client IP is added to the yellow list with a counter increased by one. All traffic associated skips TCP reassembly but undergoes single packet inspection.
2. When the client IP on the yellow list reaches a certain counter threshold, it is moved to the red list, where all associated traffic is blocked without detection.
3. A scanning thread periodically reviews the yellow list IPs. If it reaches a certain time threshold, it is removed from the yellow list.
4. A scanning thread periodically reviews the red list IPs. If it reaches a certain time threshold, it is removed from the red list and added to the yellow list.
5. Upon detecting legitimate traffic, the client IP is added to the green list with a counter increased by one. All traffic associated skips TCP reassembly but undergoes single packet inspection.
6. A scanning thread periodically reviews the green list IPs. Once a time threshold is reached, it is removed from the green list.
7. A sampling thread randomly selects IPs from the green list based on probability. All associated traffic undergoes TCP reassembly. If malicious traffic is detected, it is removed from the green list and added to the yellow list.

Sampling Algorithm

1. Poisson Sampling

Poisson sampling is a type of random sampling. Due to its resistance to synchronization issues, it can precisely measure periodic behavior and is less affected by other newly added samples. Hence, IPPM recommends Poisson sampling for network traffic sampling.

Steps to generate Poisson distribution sampling intervals:

1. Decide on the parameter λ for sampling, e.g., the average sampling interval is 30 seconds. The time unit is seconds, so λ=30, θ=1/30.
2. Generate a series of exponentially distributed (pseudo) random numbers E1, E2, …, En, … First sampling occurs at E1, the second at E1+E2, and so on. A method to generate exponentially distributed (pseudo) random numbers Ei: Generate uniformly distributed U1, U2, …, Un, … (pseudo) random numbers between 0 and 1. Use these Ui values to generate the required Ei values: Ei = -log(Ui) / θ, where log(Ui) is the natural logarithm of Ui.

2. Length-based Hierarchical Sampling

Network traffic has two notable characteristics: a) The number of short flows far exceeds the number of long flows; b) Using the same sampling probability, short flows have a much lower probability of being sampled compared to long flows. Usually, abnormal attack traffic such as DDoS, port scans, and worm propagation appears as short flows. Therefore, to ensure the effectiveness of subsequent abnormal attack traffic detection, it is necessary to increase sampling accuracy for short flows when designing a sampling algorithm.

When abnormal traffic is detected, set adaptive sampling probabilities to p1 = 0.2, p2 = 0.1, p3 = 0.01. If no abnormal traffic is detected, set a fixed sampling probability of p = 0.2.

Flow Timeout Aging Strategy

Let constants T1 and TN represent two timeout thresholds, where T1 < TN, and constant tg represents a time granularity constant, and TN-T1 is an integer multiple of tg. If using a timeout threshold set {T1, T1+tg, …, TN} for the same IP trace to perform network flow identification, it will generate a series of network flow sets, where the series of network flow numbers aligns per their respective timeout thresholds in sequence, forming a sequence {FT1, FT1+tg, …, FTN}.

For any timeout threshold t, when using it for an IP trace to perform network flow identification, the total number of generated network flows is denoted as Ft. For the following larger timeout threshold t+tg, the total number of generated network flows based on the same IP trace is denoted as Ft+tg. Then the timeout threshold t corresponds to the network flow number change rate, marked as CPFNt:

MC: The mean of the network flow number change rate set

RC: Three-quarters of the mean value MC of the network flow number change rate

A reasonable TCP network flow timeout threshold should be greater than all MC fluctuant timeout thresholds and its corresponding network flow number change rate should be less than the reasonable reference value RC. Such a timeout threshold is termed a reasonable timeout threshold and can ensure the integrity of the TCP network flow. Additionally, considering the TCP network flow identification efficiency, select the minimum value among reasonable timeout thresholds as the timeout threshold for the IP trace network traffic data, hence:

Under the current network environment, 64 seconds is an empirical value for a reasonable timeout threshold in terms of network flow identification, which can ensure a certain level of network flow completeness and identification rate.