Comprehensive Guide to IDS Development: Key Components, Industry Products, and Best Practices

1. Intrusion Detection System (IDS)

• An Intrusion Detection System (IDS) monitors network traffic for unusual or suspicious activities and sends alerts to administrators.
• Based on the source of information, IDS can be divided into Host-based IDS and Network-based IDS. Based on detection methods, it can be categorized into Anomaly-based Detection and Misuse Detection. Unlike firewalls, an IDS is a listening device that does not bridge any link and can operate without network traffic passing through it.

2. Development

• In the mid-1980s, IDS gradually evolved into the Intrusion Detection Expert System (IDES).
• In 1990, IDS split into Network-based IDS and Host-based IDS. Later, Distributed IDS emerged.
• IDS has developed rapidly, with some claiming that IDS can completely replace firewalls.

3. Classification

• Technically, intrusion detection is divided into two categories: one based on Signatures and the other based on Anomalies.
  • Signature-based detection techniques define the characteristics of events that violate security policies, such as certain header information in network packets. The detection primarily determines whether these characteristics appear in the collected data.
  • Anomaly-based detection techniques first define a set of “normal” values for the system, such as CPU utilization, memory utilization, and file checksums. Then, they compare the system’s runtime values with the defined “normal” conditions to determine if there are signs of an attack. The core of this detection method is how to define the so-called “normal” conditions.
  • The methods and conclusions of the two detection techniques differ significantly. The core of signature-based detection is maintaining a knowledge base. For known attacks, it can report the attack type in detail and accurately, but it is limited in detecting unknown attacks, and the knowledge base must be continuously updated. Anomaly-based detection cannot accurately identify the attack method, but it can detect a broader range of attacks, including those not yet discovered.

4. Components

• Event Generators: Their purpose is to obtain events from the entire computing environment and provide these events to other parts of the system.
• Event Analyzers: They analyze the obtained data and produce analysis results.
• Response Units: These functional units respond to the analysis results, which can include actions like disconnecting connections, changing file attributes, or simply raising an alert.
• Event Databases: These are places where various intermediate and final data are stored. They can be complex databases or simple text files.

5. Intrusion Detection Products in the Industry

• SolarWinds Security Event Manager: Analyzes logs from Windows, Unix, Linux, and Mac OS systems. It manages data collected by Snort, including real-time data. SEM is also an intrusion prevention system with over 700 rules to shut down malicious activities. An important tool for improving security, responding to incidents, and achieving compliance.
• CrowdStrike Falcon (Free Trial): A cloud-based endpoint protection platform that includes threat hunting.
• ManageEngine EventLog Analyzer (Free Trial): A log file analyzer used to search for evidence of intrusions.
• Snort: Provided by Cisco Systems, free to use, a leading network-based intrusion detection system software.
• OSSEC: An excellent host-based intrusion detection system, free to use.
• Suricata: Network-based intrusion detection system software that operates at the application layer to improve visibility.
• Zeek: A network monitor and network-based intrusion prevention system.
• Sagan: A log analysis tool that can integrate reports generated from Snort data, making it a HIDS with a small amount of NIDS.
• Security Onion: A network monitoring and security tool composed of elements extracted from other free tools.
• AIDE: Advanced Intrusion Detection Environment, a HIDS for Unix, Linux, and Mac OS.
• OpenWIPS-NG: A wireless NIDS and intrusion prevention system from the makers of Aircrack-NG.
• Samhain: A direct host-based intrusion detection system for Unix, Linux, and Mac OS.
• Fail2Ban: A lightweight host-based intrusion detection software system for Unix, Linux, and Mac OS.

6. Host-based Intrusion Detection

• Host-based intrusion detection systems check events on the computer rather than the traffic passing around the system. This type of intrusion detection system is abbreviated as HIDS and primarily operates by viewing data in management files on the protected computer. These files include log files and configuration files.
• HIDS will back up your configuration files so that you can restore settings if a malicious virus compromises system security by changing the computer’s settings. Another key factor to prevent is root user access on Unix-like platforms or registry changes on Windows systems. HIDS will not stop these changes but should alert you when any such access occurs.
• Some software must be installed on each host monitored by HIDS. You can have HIDS monitor just one computer, but it is more common to install HIDS on every device on the network. This is because you do not want to overlook configuration changes on any device. Naturally, if you have multiple HIDS hosts on your network, you do not need to log into each HIDS host to get feedback. Therefore, a distributed HIDS system needs to include a centralized control module. Look for a system that encrypts communication between the host agents and the central monitor.

7. Network-based Intrusion Detection

• Network-based intrusion detection examines traffic on the network. Therefore, a typical NIDS must include a packet sniffer to collect network traffic for analysis.
• The analysis engine of NIDS is usually rule-based and can be modified by adding your own rules. For many NIDS, the system provider or user community will provide rules that you can import into your implementation. Once familiar with the rule syntax of your chosen NIDS, you can create your own rules.
• Linking back to traffic collection, you do not want to dump all traffic into a file or run everything through a dashboard because you will not be able to analyze all that data. Therefore, the rules driving analysis in NIDS also create selective data capture. For example, if you have rules about concerning types of HTTP traffic, the NIDS should only select and store HTTP packets showing those characteristics.
• Typically, NIDS is installed on dedicated hardware. High-end paid enterprise solutions are provided as network suites with pre-installed software. However, you do not need to spend a lot of money to get professional hardware. NIDS does need a sensor module to receive traffic, so you can load it onto a LAN analyzer or choose to allocate a computer to run the task. However, ensure the device chosen for the task has sufficient clock speed to avoid slowing down the network.

8. Summary

• 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

  IDS	HIDS/NIDS	Unix	Linux	Windows	Mac OS
  SolarWinds Security Event Manager	Both	No	No	Yes	No
  CrowdStrike Falcon	HIDS	Yes	Yes	Yes	Yes
  Snort	NIDS	Yes	Yes	Yes	No
  OSSEC	HIDS	Yes	Yes	Yes	Yes
  ManageEngine EventLog Analyzer	HIDS	Yes	Yes	Yes	Yes
  Suricata	NIDS	Yes	Yes	Yes	Yes
  Zeek	NIDS	Yes	Yes	No	Yes
  Sagan	Both	Yes	Yes	No	Yes
  Security Onion	Both	No	Yes	No	No
  AIDE	HIDS	Yes	Yes	No	Yes
  Open WIPS-NG	NIDS	No	Yes	No	No
  Samhain	HIDS	Yes	Yes	No	Yes
  Fail2Ban	HIDS	Yes	Yes	No	Yes

  
  
• 
  

  SolarWinds Security Event Manager: Runs on Windows Server but can log messages generated by Unix, Linux, and Mac OS computers as well as Windows PCs. As a log manager, it is a host-based intrusion detection system because it deals with files on the management system. However, it also manages data collected by Snort, making it part of a network-based intrusion detection system.

  
  

  Snort is a widely used packet sniffer created by Cisco Systems. It has a specific data format that other IDS tool manufacturers have integrated into their products. This is the case with SolarWinds Security Event Manager. A network intrusion detection system examines how traffic data propagates across the network. To deploy the NIDS capabilities of Security Event Manager, you need to use Snort as the packet capture tool and funnel the captured data into Security Event Manager for analysis. Although LEM can act as a HIDS tool when handling log file creation and integrity, it can receive real-time network data through Snort, which is an NIDS activity.

  
  

  SolarWinds products can also act as intrusion prevention systems because they can trigger measures to detect intrusions. The software package comes with over 700 event correlation rules, enabling it to detect suspicious activities and automatically implement remedial actions. These actions are

  
  

  Falcon Insight logs events on protected computers and needs to store them in log files. Once these events are written, the tool’s research and detection elements use a pure HIDS strategy. The event collection element of the EPP is the agent, which must be installed on the protected device. This agent communicates with the EPP’s central processing system, which resides in the cloud. Human administrators of protected endpoints can access the Falcon dashboard through any standard browser.

  
  

  The advantage of CrowdStrike Falcon software’s hybrid local/cloud architecture is that the system is very lightweight on your device. The analysis software on CrowdStrike’s servers provides all the processing power needed for threat analysis. This means that installing this security service will not slow down your computer, allowing it to perform the tasks it is intended for. However, the agent also acts as a threat remediation implementer, so it can continue to work even if the Internet connection is unavailable.

  
  
• 
  

  Snort is the industry leader in NIDS and is still available for free. It is one of the few IDS that can be installed on Windows. It was created by Cisco. The system can operate in three different modes and can implement defense strategies, making it both an intrusion prevention system and an intrusion detection system. Snort’s three modes are:

  
  

  Sniffer Mode, Packet Logger, Intrusion Detection

  
  

  You can use Snort as a packet sniffer without enabling its intrusion detection capabilities. In this mode, you can read packets passing through the network in real-time. In packet logging mode, those packet details are written to a file. When you access Snort’s intrusion detection capabilities, you invoke an analysis module that applies a set of rules to the passing traffic. These rules are called “base policies,” and if you don’t know which rules you need, you can download them from the Snort website. However, once you are confident in Snort’s methodology, you can write your own. This IDS has a large community base that is very active on the Snort website’s community pages. You can get tips and help from other users and download rules that experienced Snort users have created.

  
  
• 
  

  OSSEC stands for Open Source HIDS Security. It is the leading HIDS and is completely free to use. As a host-based intrusion detection system, this program focuses on the log files on the computer it is installed on. It monitors the checksums and signatures of all log files to detect possible tampering. On Windows, it keeps tabs on any changes made to the registry. On Unix-like systems, it monitors any attempts to access the root account. Although OSSEC is an open-source project, it is actually owned by Trend Micro, a well-known security software producer.

  
  

  The main monitoring application can cover one computer or multiple hosts, consolidating data into one console. Although there is a Windows agent to monitor Windows computers, the main application can only be installed on Unix-like systems, meaning Unix, Linux, or Mac OS. The main program has an interface for OSSEC, but this interface is installed separately and is no longer supported. Regular OSSEC users have found other applications that work well as front ends for data collection tools: Splunk, Kibana, and Graylog.

  
  

  The log files covered by OSSEC include FTP, mail, and web server data. It also monitors operating system event logs, firewall and antivirus logs and tables, and traffic logs. OSSEC’s behavior is controlled by the policies you install on it. These add-ons can be obtained from the product’s active, large user community. Policies define alert conditions. These alerts can be displayed on the console or sent as notifications via email. Main features: Log file analyzer, free policies, alert system

  
  
• 
  

  Suricata may be the main alternative to Snort. Suricata has a key advantage over Snort in that it collects data at the application layer. This overcomes Snort’s blindness to signatures split across multiple TCP packets. Suricata waits for all the data in the packets to be assembled before moving the information into analysis.

  
  

  Although the system works at the application layer, it can monitor lower-level protocol activities such as IP, TLS, ICMP, TCP, and UDP. It checks real-time traffic for different network applications, including FTP, HTTP, and SMB. The monitor not only looks at the structure of the packets. It can check TLS certificates and focus on HTTP requests and DNS calls. The file extraction tool allows you to inspect and isolate suspicious files that have characteristics of virus infections.

  
  

  Suricata is compatible with Snort, and you can use the same VRT rules written for that NIDS leader. Third-party tools integrated with Snort, such as Snorby, BASE, Squil, and Anaval, can also connect to Suricata. Therefore, accessing the Snort community for tips and free rules can be a great benefit for Suricata users. The built-in scripting module allows you to combine rules and achieve more precise detection profiles than Snort can offer. Suricata uses both signature and anomaly detection methods.

  
  

  Suricata has a clever processing architecture that achieves hardware acceleration by performing multithreaded activities simultaneously using many different processors. It can even partially run on your graphics card. This distribution of tasks avoids the load being borne by just one host. This is good because one issue with this NIDS is its high processing volume.

  
  
• 
  

  Zeek is a free NIDS that not only performs intrusion detection but also provides other network monitoring functions. Zeek’s user community includes many academic and research institutions. Zeek’s intrusion detection functions are completed in two stages: Traffic Logging and Analysis. Like Suricata, Zeek has a major advantage over Snort because its analysis is performed at the application layer. This allows you to view data across packets for a broader analysis of network protocol activity.

  
  

  Zeek’s analysis module has two elements, allowing for both signature detection and anomaly analysis. The first of these analysis tools is the Zeek event engine. This tracks triggering events, such as new TCP connections or HTTP requests. Each event is logged, so this part of the system is policy-neutral—it simply provides a list of events where analysis might reveal repeated actions or suspicious activities generated by the same user account.

  
  

  The mining of this event data is performed by policy scripts. Alert conditions trigger actions, making Zeek an intrusion prevention system as well as a network traffic analyzer. Policy scripts can be customized, but they usually follow a standard framework involving signature matching, anomaly detection, and connection analysis.

  
  

  You can use Zeek to track HTTP, DNS, and FTP activities, and also monitor SNMP traffic, allowing you to check device configuration changes and SNMP trap conditions. Each policy is a set of rules, and you are not limited to the number of active policies or the other layers of the protocol stack you can check. At lower levels, you can watch out for DDoS Syn Flood attacks and detect port scans.

  
  
• 
  

  Sagan is a host-based intrusion detection system, making it an alternative to OSSEC and available for free. Although it is a HIDS, this program is compatible with data collected by the NIDS system Snort. This compatibility extends to other tools that can be used with Snort, such as Snorby, BASE, Squil, and Anaval. Data sources from Zeek and Suricata can also be input into Sagan. The tool can be installed on Unix, Linux, and Mac OS. Although you cannot run Sagan on Windows, you can input Windows event logs into it.

  
  

  Strictly speaking, Sagan is a log analysis tool. What it lacks to be a standalone NIDS is a packet sniffer module. However, on the plus side, this means Sagan does not require dedicated hardware and can flexibly analyze host logs and network traffic data. The tool must be used in conjunction with other data collection systems to create a complete intrusion detection system.

  
  

  Some nice features of Sagan include an IP locator, which allows you to see the geographic location of IP addresses detected as having suspicious activity. This enables you to correlate the actions of IP addresses that appear to be working in concert, forming an attack. Sagan can distribute its processing across multiple devices, reducing the load on the key server’s CPU.

  
  

  The system includes script execution, meaning it will generate alerts and take action when an intrusion scenario is detected. If suspicious activity is detected from a specific source, it can interact with firewall tables to implement IP bans. Therefore, it is an intrusion prevention system. The analysis module can be used for signature and anomaly detection methods.

  
  
• 
  

  For IDS solutions, you can try the free Security Onion system. Most of the IDS tools in this list are open-source projects. This means anyone can download the source code and make changes. This is exactly what the developers of Security Onion have done. They have taken elements from the source code of Snort, Suricata, OSSEC, and Zeek and stitched them together to create a free Linux-based NIDS/HIDS hybrid. Security Onion is written to run on Ubuntu and also integrates elements from front-end systems and analysis tools, including Snorby, Sguil, Squit, Kibana, ELSA, Xplico, and NetworkMiner.

  
  

  Although Security Onion is classified as a NIDS, it does include HIDS capabilities. It will monitor your logs and configuration files for suspicious activity and check for any unexpected changes in the checksums of these files. One downside of Security Onion’s comprehensive network infrastructure monitoring approach is its complexity. It has several different operational structures and does not have enough online learning resources or bundled learning materials to help network administrators master all the tool’s features.

  
  

  Network analysis is performed by a packet sniffer, which can display passing data on the screen or write it to a file. Security Onion’s analysis engine complicates things because there are many different tools with different operational procedures, and you may end up ignoring most of them. Kibana’s interface provides the dashboard for Security Onion and does include some nice graphics and charts to simplify status recognition.

  
  
• 
  

  Produced by Germany’s Samhain Design Labs, Samhain is free host-based intrusion detection system software. It can run on a single computer or multiple hosts, providing centralized data on events detected by the agents running on each computer.

  
  

  Tasks performed by each agent include file integrity checking, log file monitoring, and port monitoring. Processes look for rootkit viruses, rogue SUID (user access permissions), and hidden processes. In multi-host implementations, the system applies encryption to communications between agents and the central controller. Connections that transmit log file data include authentication requirements to prevent intruders from hijacking or replacing the monitoring process.

  
  

  The data collected by Samhain can analyze activities on the network and highlight warning signs of intrusions. However, it does not prevent intrusions or clean up malicious processes. You will need to maintain backups of configuration files and user identities to resolve issues highlighted by the Samhain monitor.

  
  

  One problem with hacker and virus intrusions is that intruders will take steps to hide their presence. This includes terminating monitoring processes. Samhain deploys a stealth technique to hide its processes, preventing intruders from manipulating or killing the IDS. This stealth method is called “steganography.”

  
  

  Central log files and configuration backups are signed with PGP keys to prevent tampering by intruders.

  
  

  Samhain is an open-source network intrusion detection system available for free download. It is designed according to POSIX guidelines to be compatible with Unix, Linux, and Mac OS. The central monitor consolidates data from different operating systems.

  
  
• 
  

  Fail2Ban is a free host-based intrusion detection system focused on detecting worrying events recorded in log files, such as excessive failed login attempts. The system sets blocks on IP addresses showing suspicious behavior. These bans usually last only a few minutes, but this is enough to disrupt standard automated brute-force password cracking attempts. This security strategy can also be effective against DoS attacks. The actual length of IP address bans can be adjusted by the administrator.

  
  

  Fail2Ban is actually an intrusion prevention system because it takes action when suspicious activity is detected, rather than just recording and highlighting possible suspicious intrusions.

  
  

  Therefore, system administrators must be careful with access policies when setting up the software, as overly strict prevention policies can easily lock out legitimate users. The problem with Fail2Good news is, all the systems we listed are either free or offer free trials, so you can try out some of them. The user community aspect of these systems might be particularly appealing to you, especially if you already have a colleague with extensive experience. The ability to get tips from other network administrators is definitely an attraction of these systems. It’s even more appealing compared to paid solutions with professional help desk support.

  
  

  If your company’s industry requires standard security compliance, such as PCI, then you definitely need a proper IDS solution. Additionally, if you keep personal information in the public domain, your data protection program must be strictly enforced to prevent your company from being sued due to data breaches.