Optimizing Suricata with Hyperscan: Enabling High-Performance Regex Matching

What is Suricata, including the keyword Hyperscan?

• Suricata is a free, open-source, mature, high-performance, stable network threat detection engine
• System functionalities include: real-time intrusion detection (IDS), inline intrusion prevention (IPS), network security monitoring (NSM), and offline pcap processing
• Suricata relies on a powerful and extensible rules and signature language to filter network traffic, and supports LUA scripting language
• The output file formats are YAML or JSON, making it easy to integrate with other databases or security data analysis platforms
• Suricata is community-driven in its development, facilitating version maintenance and iteration of new features

Features

• IDS / IPS

A comprehensive signature language is used to describe known threats and malicious behaviors,
compatible with the Emerging Threats Suricata ruleset (Proofpoint and Intel rules)
and VRT ruleset (Snort rules), supporting Barnyard and Barnyard2 tools

• High Performance

A single instance of Suricata can detect gigabit network traffic; the engine is based on multithreading and hardware acceleration (pf_ring, af_packet)

• Automatic Protocol Detection

Automatic protocol scanning on ports, conducive to discovering malware and communication channels

• NSM: More than an IDS

Suricata can log all HTTP request links, DNS requests, and TLS key exchanges, and supports extracting and storing information from streams to disk.

• Lua Scripting

LUA scripts can compensate for characteristics that cannot be described by the rule set

• Industry Standard Outputs

The primary log output format is Eve, which includes all protocol events and alert outputs
(can specify hosts or subnet segments separately, configure global rules or individual rules), traffic logs

• Operating System

Linux, FreeBSD, OpenBSD, macOS / Mac OS X, Windows

• Configuration

YAML as the rule file format, known for its readability

• TCP/IP Engine

Supports IPv6 and tunnel decoding including: Teredo, IP-IP, IP6-IP4, IP4-IP6, GRE,
VXLAN, Geneve, supports session tracking and stream reassembly, IP fragmentation reassembly,
supports a variety of protocol decoding including IPv4, IPv6, TCP, UDP, SCTP, ICMPv4, ICMPv6, GRE, Ethernet,
PPP, PPPoE, Raw, SLL, VLAN, QINQ, MPLS, ERSPAN, VXLAN, Geneve, HTTP, HTTP/2,
SSL, TLS, SMB, DCERPC, SMTP, FTP, SSH, DNS, Modbus, ENIP/CIP, DNP3, NFS, NTP,
DHCP, TFTP, KRB5, IKEv2, SIP, SNMP, RDP, RFB, MQTT

• HTTP Engine

Implements a stateful HTTP parser based on libhtp, capable of parsing URLs, request and response headers, cookies,
user-agent, request body, and response body, request methods, and status codes, host

• Detection Engine

Can perform detection matching based on multiple characteristics, including: protocol keywords, regular expressions (Hyperscan),
fast pattern and preprocessor, file matching, JA3 / JA3S / HASSH matching.
Rules support real-time loading without restarting Suricata, and rule lazy initialization

• Packet Acquire

High-performance capture modes: AF_PACKET, PF_RING, NETMAP
Standard modes: NFLOG, PCAP
IPS modes: Netfilter, NETMAP, AF_PACKET (Linux), ipfw (FreeBSD and NetBSD)

• Multi-Threading

Thread support is configurable, ranging from one to dozens; mutex operations use atomic functions to improve performance;

Architecture

• Packet Capture
  AF_PACKET and PF_RING use flow (5-tuple) symmetric hashing to threads
  RSS technology distributes traffic by assigning it to different queues on the NIC, but the downside is that asymmetric encryption can lead to errors in two-way traffic detection like TCP. Therefore, we usually configure one RSS queue or use a symmetric hashing algorithm, and disable NIC offloading, as offloading can cause the inability to track certain stream states. Here’s a configuration example:
  PF_RING: 1 RSS queue and use cluster-type ‘cluster_flflow’. Disable NIC offloading except the rx/tx checksum
• Hyperscan

Hyperscan is a high-performance multiple regex matching library. In Suricata, it can be used to perform multiple pattern matching (mpm).

First, install Hyperscan:
apt-get install cmake ragel

apt-get install libboost-dev

sudo apt-get python-dev libbz2-dev
wget https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.tar.gz
tar xvzf boost_1_66_0.tar.gz
cd boost_1_66_0
./bootstrap.sh --prefix=~/tmp/boost-1.66
./b2 install

git clone https://github.com/intel/hyperscan
cd hyperscan
mkdir build
cd build
cmake -DBUILD_STATIC_AND_SHARED=1 ../

cmake -DBUILD_STATIC_AND_SHARED=1 -DBOOST_ROOT=~/tmp/boost-1.66 ../

make
sudo make install

Next, by default, Suricata does not have Hyperscan enabled at compile time. We need to explicitly add the following commands when compiling Suricata:

–with-libhs-includes=/usr/local/include/hs/ –with-libhs-libraries=/usr/local/lib/
Then modify the mpm-algo and spm-algo values to ‘hs’ in suricata.yaml

High Performance Configuration
If you have enough RAM, consider the following options in suricata.yaml
detect:
profile: custom
custom-values:
toclient-groups: 200
toserver-groups: 200
sgh-mpm-context: auto
inspection-recursion-limit: 3000

• Statistics
  Writes packet data to the stats.log file every 8 seconds:

When Suricata is closed, you can also see the total number of packets sent and received, and packet loss

The numbers displayed may vary with different capture modes:

In AF_PACKET mode:
• kernel_packets is the number of packets correctly sent to userspace
• kernel_drops is the number of packets that have been discarded instead of being sent to userspace
In PF_RING mode:
• kernel_packets is the total number of packets seen by pf_ring
• kernel_drops is the number of packets that have been discarded instead of being sent to userspace

• Ignoring Traffic

1. You can ignore certain types of traffic by configuring a bpf file
   echo “not host 1.2.3.4” > capture-filter.bpf
   suricata -i ens5f0 -F capture-filter.bpf
   Syntax similar to: “not (host IP1 or IP2 or IP3 or net NET/24)” or “ tcp or udp”
2. Or you can also use Suricata rules:
   pass/drop ip 1.2.3.4 any <> any any (msg:”pass all traffic from/to 1.2.3.4″; sid:1;)
3. Set the tls keyword to pass all traffic after the HTTPS handshake without detection:
   app-layer.protocols.tls.encryption-handling