IDS and IPS with Snort 3 E-Book

Ashley Thomas

• Herausgeber: Packt Publishing
• Kategorie: Fachliteratur
• Sprache: Englisch

Seitenzahl: 352

Veröffentlichungsjahr: 2024

Beschreibung

Snort, an open source intrusion detection and prevention system (IDS/IPS), capable of real-time traffic analysis and packet logging, is regarded as the gold standard in IDS and IPS. The new version, Snort 3, is a major upgrade to the Snort IDS/IPS, featuring a new design and enhanced detection functionality, resulting in higher efficacy and improved performance, scalability, usability, and extensibility. Snort 3 is the latest version of Snort, with the current version at the time of writing being Snort v3.3.3.
This book will help you understand the fundamentals of packet inspection in Snort and familiarize you with the various components of Snort. The chapters take you through the installation and configuration of Snort, focusing on helping you fine-tune your installation to optimize Snort performance. You’ll get to grips with creating and modifying Snort rules, fine-tuning specific modules, deploying and configuring, as well as troubleshooting Snort. The examples in this book enable network administrators to understand the real-world application of Snort, while familiarizing them with the functionality and configuration aspects.
By the end of this book, you’ll be well-equipped to leverage Snort to improve the security posture of even the largest and most complex networks.

Leseprobe

IDS and IPS with Snort 3

Get up and running with Snort 3 and discover effective solutions to your security issues

Ashley Thomas

IDS and IPS with Snort 3

Copyright © 2024 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Group Product Manager: Pavan Ramchandani

Publishing Product Manager: Neha Sharma

Book Project Manager: Ashwin Dinesh Kharwa

Senior Editor: Apramit Bhattacharya

Technical Editor: Irfa Ansari

Copy Editor: Safis Editing

Proofreader: Apramit Bhattacharya

Indexer: Subalakshmi Govindhan

Production Designer: Vijay Kamble

DevRel Marketing Coordinator: Kamalpreet Kaur Sahni

First published: September 2024

Production reference: 1040924

Published by Packt Publishing Ltd.

Grosvenor House

11 St Paul’s Square

Birmingham

B3 1RB, UK

ISBN 978-1-80056-616-3

www.packtpub.com

To my son, Rohan.

Contributors

About the author

Ashley Thomas is a security researcher at Dell SecureWorks and a member of the Counter Threat Unit team. Before this role, he was instrumental in building the iSensor, a proprietary network intrusion prevention system. Ashley has a master’s in computer networking from North Carolina State University, and he also holds several other certifications, including CISSP, GCIA, GREM, GCLD, and GWEB. He has authored several papers on intrusion detection and holds several patents in this field.

About the reviewers

Ron Cowen has been in the network security industry for over a decade, spanning roles at AT&T, Juniper Networks, and his current position as a senior systems engineer for Palo Alto Networks. He is based in Seattle, Washington.

I’d like to acknowledge and thank all of those who have supported, and those who continue to support, my growth as a network security professional, as well as my wife and our two daughters

Wayne Burke, VP for Cyber2labs.com, is internationally recognized for his commitment, achievements, and contributions to the IT security industry. He currently specializes in many offensive and defensive AI technologies for robotics such as drones, building and managing new high-tech security tools, custom hardware solutions for bio-medical products, digital forensics, penetration testing, and mobile security, and radio-frequency SDRs. Wayne and his team have delivered security assessments, penetration test assignments, and customized training for international corporations and many government agencies, such as EPA, FAA, DOJ, DOE, DOD, the Air Force, the Army, the Navy, the Marines, CIA, FBI, NSA, and many more US government bodies.

Table of Contents

Preface

Part 1: The Background

1

Introduction to Intrusion Detection and Prevention

The need for information security

Defense-in-depth strategy

Firewalls (network and host layers)

Intrusion detection and prevention systems (network and host layers)

Endpoint detection and response (host layer)

Web application firewalls (network and host layers)

Mail security gateway (network)

Log management and monitoring (network and host)

The role of network IDS and IPS

Types of intrusion detection

Signature-based intrusion detection

Anomaly-based intrusion detection

Hybrid intrusion detection

The state of the art in IDS/IPS

Stateful analysis

Fast packet acquisition

Parallel processing

Pattern matching

Extending rule language

App and protocol identification

File analysis

IDS/IPS metrics

Detection accuracy

Performance-related IDS/IPS metrics

IDS/IPS evaluation and comparison

Evasions and attacks

IDS/IPS evasions

Attacks against the IDS/IPS

Summary

2

The History and Evolution of Snort

The beginning of Snort

Snort 1 – key features and limitations

Snort 2 – key features, improvements, and limitations

Snort 2.9

The need for Snort 3

Summary

Part 2: Snort 3 – The New Horizon

3

Snort 3 – System Architecture and Functionality

Design goals

High performance

Pluggable modular architecture

Configurability and customizability

Efficiency

Key components

DAQ module

Codecs

Inspectors

Detection or rule engine

Configuration module

Alerting and logging module

Snort 3 system architecture

Multithreading

Packet analysis flow within each Snort thread

Summary

4

Installing Snort 3

Choosing an OS for installing Snort 3

Snort 3 installation process

Preparing the system

Installing dependencies

Installing Snort 3

Installing Snort 3 on CentOS

Preparing the system

Installing build tools

Installing dependencies

Installing Snort 3

Installing Snort 3 on Kali (Debian)

Preparing the system

Installing dependencies

Installing Snort 3

Summary

5

Configuring Snort 3

Configuring Snort 3 – how?

Command-line arguments

Configuration files

Configuring Snort 3 – what?

Configuring defaults

Configuring inspection

Configuring bindings

Configuring performance

Configuring detection

Configuring filters

Configuring output

Configuring your environment

HOME_NET

EXTERNAL_NET

HTTP_PORTS

The stream_tcp inspector

Optimal configuration and tuning

Managing multiple policies and configurations

Summary

Part 3: Snort 3 Packet Analysis

6

Data Acquisition

The functionality of the DAQ layer

The performance of the DAQ Layer

Factors affecting packet capture performance

The consequence of packet capture performance degradation

Packet capture in Snort

Before DAQ

The DAQ module – introduced in Snort 2.9

The Snort 3 implementation of the DAQ layer

The DAQ library API

DAQ modules

Configuring DAQ

Summary

7

Packet Decoding

OSI layering and packet structure

Data encapsulation and decapsulation

The role of packet decoding (Codecs)

Packet decoding in Snort 3

EthCodec – a layer 2 codec

IPv4Codec – a layer 3 codec

TcpCodec – a layer 4 codec

Code structure and other codecs

Summary

8

Inspectors

The role of inspectors

Types of inspectors

Network inspectors

Service inspectors

Stream inspectors

Snort 3 inspectors

Network inspectors

Service inspectors

Stream inspectors

Wizard and binder inspectors

Summary

9

Stream Inspectors

Relevant protocols for the stream inspector

IP

ICMP

TCP

UDP

Flow

The stream inspectors

stream_ip

stream_udp

stream_icmp

stream_tcp

stream_base

Summary

10

HTTP Inspector

Basics of HTTP

HTTP request

HTTP response

HTTP/2

HTTP inspector

HTTP buffers

HTTP/2 inspector

HTTP inspector configuration

Summary

11

DCE/RPC Inspectors

A DCE/RPC overview

Connectionless versus connection-oriented DCE/RPC

DCE/RPC inspectors

DCE/RPC rule options

Exercise

Summary

12

IP Reputation

Background

IP address as an entity – use of blocklists

Challenges

History of IP blocking in Snort

Configuration of the IP reputation inspector module

Functionality of the IP reputation inspector

Data structure for storing IP reputation scores

IP reputation inspector – alerts and pegs

Summary

Part 4: Rules and Alerting

13

Rules

Snort rule – the structure

Service rule

File rule

File identification rule

Rule header

Traditional rule header

Rule options

General rule options

Payload options

Non-payload options

Recommendations for writing good rules

Using fast_pattern wisely

Using the inspection buffers for rule matching

Defining the right service or protocol

Summary

14

Alert Subsystem

Post-inspection processing

Event generation

Event thresholding

Applying a rule action to a packet

Logging the alert

Alert formats

CSV format

Unified2 format

Alert Fast format

Alert Full format

JSON format

Summary

15

OpenAppID

The OpenAppID feature

Design and architecture

Detectors

The inspector

The rules

Exercise

Summary

16

Miscellaneous Topics on Snort 3

Snort 2 to Snort 3 migration

Migrating the rules

Migrating configurations

Troubleshooting Snort 3

Why is the Snort rule for XYZ not alerting?

Snort is crashing!

Help! Got support?

Summary

Index

Other Books You May Enjoy

Preface

Snort is recognized as the industry standard for intrusion detection and prevention systems; Snort 3 is the latest version of the software and includes significant changes to its functionality and features. This book will introduce you to IDS/IPS systems and the Snort IDS/IPS system. It will provide you with details on the latest version of Snort, Snort 3, and familiarize you with its workings and its configuration.

Who this book is for

This book is for anyone who wants to learn about Snort 3. If you are a beginner to the world of Snort, or if you are someone who has used Snort and would like to learn about its latest version, this book is for you.

We expect that network administrators, security administrators, security consultants, and other security professionals will find this book useful. Those using other intrusion detection systems (IDS) will also gain from this book as it covers the basic inner workings of any IDS. Although there are no prerequisites, basic familiarity with Linux systems and knowledge of basic network packet analysis will be very helpful.

What this book covers

Chapter 1, Introduction to Intrusion Detection and Prevention, discusses a defense-in-depth strategy and the role of various security tools, including IDS/IPS.

Chapter 2, The History and Evolution of Snort, explores the evolution of Snort from its original version to its current state. We will look at the key features of Snort and when they were incorporated into the system.

Chapter 3, Snort 3 – System Architecture and Functionality, explores the design goals, the main components, and the system architecture of Snort 3. The chapter provides you with a high-level idea of how network traffic gets analyzed by the Snort system.

Chapter 4, Installing Snort 3, shows you how to install the Snort 3 system. The chapter describes the step-by-step installation process of Snort 3 on two different operating systems.

Chapter 5, Configuring Snort 3, explains how to configure the Snort 3 system. It discusses how a user can configure the Snort 3 system and the various modules, using command-line arguments as well as configuration files.

Chapter 6, Data Acquisition, delves into the data acquisition layer and its role in the delivery and transmission of network packets to and from Snort.

Chapter 7, Packet Decoding, reinforces the idea that an analysis of network traffic begins with packet decoding. This chapter explains the process of packet decoding and discusses how the packet decoding module is structured, what the important data structures are, and how the module ties to the rest of the Snort system.

Chapter 8, Inspectors, discusses inspectors, which are considered the backbone of Snort 3 from a functionality perspective. From an evolution standpoint, the inspectors replaced the preprocessor module in Snort 2. This chapter discusses the role and functionality of the Inspector modules.

Chapter 9, Stream Inspectors, discusses the stateful analysis capability of Snort 3. The chapter also explains important terms such as flows, sessions, and streams, which are relevant to how Snort performs stateful analysis.

Chapter 10, HTTP Inspector, explores HTTP, which is one of the most prevalent protocols used over the internet. This chapter discusses the HTTP inspector and how it enables the detection of malicious attacks over the HTTP protocol.

Chapter 11, DCE/RPC Inspectors, discusses the DCE/RPC inspectors and their overview, dependencies, relevant rule options, and configurations.

Chapter 12, IP Reputation, shows you how the IP reputation inspector module works, its configuration, and its importance.

Chapter 13, Rules, discusses how Snort rules work, its structure, and some important points to keep in mind while developing Snort rules. The use of Snort rules allows a Snort user to specify what constitutes malicious traffic.

Chapter 14, Alert Subsystem, delves into the alert subsystem of Snort. We will discuss the various alert modules and how they are configured.

Chapter 15, OpenAppID, discusses the OpenAppID feature, the relevant inspector modules, and their configuration.

Chapter 16, Miscellaneous Topics on Snort 3, discusses a handful of miscellaneous topics related to Snort 3. We will explore how to go about troubleshooting and/or debugging Snort, Snort 2 to Snort 3 migration challenges, and so on.

To get the most out of this book

You are expected to know the basics of computer networking, networking protocols, and traffic analysis. Familiarity with network traffic analysis tools such as Wireshark and/or tcpdump will be useful. Familiarity with Linux operating systems is also expected.

Software/hardware covered in the book

Operating system requirements

Snort 3

Linux

If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code.

Download the example code files

You can download the example code files for this book from GitHub at https://github.com/PacktPublishing/IDS-and-IPS-with-Snort-3.0. If there’s an update to the code, it will be updated in the GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Conventions used

There are a number of text conventions used throughout this book.

Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “Another key component that was included in this release was the IP defrag module.”

A block of code is set as follows:

alert tcp any any -> $HOME_NET [80,8080] (msg:"SQL Injection Detected"; flow:established,to_server; http_uri; content:"/wordpress/wp-content/plugins/demo_vul/endpoint.php"; content:"union"; distance:0; http_uri; content:"select"; distance:0; nocase; content:"from"; distance:0; nocase; sid:123;)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

http://acunetix.php.example/wordpress/wp-content/plugins/demo_vul/endpoint.php?user=-1+union+select+1,2,3,4,5,6,7,8,9,(SELECT+user_pass+FROM+wp_users+WHERE+ID=1)

Any command-line input or output is written as follows:

sudo dnf install -y flex bison gcc gcc-c++ make cmake automake autoconf libtool curl pkgconf

Bold: Indicates a new term, an important word, or words that you see on screen. For instance, words in menus or dialog boxes appear in bold. Here is an example: “It can be noted that the Total Length field is 16 bits.”

Tips or important notes

Appear like this.

Get in touch

Feedback from our readers is always welcome.

General feedback: If you have questions about any aspect of this book, email us at [email protected] and mention the book title in the subject of your message.

Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form.

Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at [email protected] with a link to the material.

If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

Share Your Thoughts

Once you’ve read IDS and IPS with Snort 3, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback.

Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

Download a free PDF copy of this book

Thanks for purchasing this book!

Do you like to read on the go but are unable to carry your print books everywhere?

Is your eBook purchase not compatible with the device of your choice?

Don’t worry, now with every Packt book you get a DRM-free PDF version of that book at no cost.

Read anywhere, any place, on any device. Search, copy, and paste code from your favorite technical books directly into your application.

The perks don’t stop there, you can get exclusive access to discounts, newsletters, and great free content in your inbox daily

Follow these simple steps to get the benefits:

Scan the QR code or visit the link below

https://packt.link/free-ebook/9781800566163

Submit your proof of purchaseThat’s it! We’ll send your free PDF and other benefits to your email directly

Part 1: The Background

Information security plays a crucial role in the successful operation of any organization. Intrusion Detection Systems (IDS) and Intrusion Prevention Systems (IPS) have a pivotal role in a defense-in-depth information security strategy. One of the leading open source IDS/IPS systems of our time is Snort.

The first part of the book covers the necessary background information about network security and intrusion detection, the role of information security, and the role of an IDS/IPS system within a defense-in-depth strategy. A brief history of the evolution of Snort to its current state is also provided. With this background, we will start discussing Snort 3 in the second part of the book.

This part has the following chapters:

Chapter 1, Introduction to Intrusion Detection and PreventionChapter 2, The History and Evolution of Snort

1

Introduction to Intrusion Detection and Prevention

Information security plays a key role in the successful operation of any organization; it ensures the confidentiality, integrity, and availability of information. Intrusion detection systems (IDS) and intrusion prevention systems (IPS) play a critical role in the defense-in-depth strategy used in the information security field. Historically, the role of intrusion detection was primarily that of monitoring in order to detect malicious or suspicious activity. Over time, the prevention capability was added in addition to detection, thereby creating IPS. As the nature of computation evolved over time, the nature of threat and attack vectors also evolved. Subsequently, the complexity of analysis and computation required by intrusion detection has also evolved in order to address the threat landscape. This chapter will introduce you to IDS and IPS at a high level. The chapter will cover the following topics:

The need for information securityDefense-in-depth strategyThe role of network IDS and IPSTypes of intrusion detectionThe state of the art in IDS/IPSIDS/IPS metricsEvasions and attacks

The need for information security

Software and IT are everywhere, and their adoption is increasing at an ever-increasing speed. Software programming is prevalent in the fields of entertainment, health, education, food, travel, auto, communication, media, and every other field we can think of. As the number of software programs and their features increase, so does the number of software bugs and flaws. A security flaw, glitch, or weakness found in software code that could be exploited by an attacker (threat source) is called a software vulnerability. The number of such vulnerabilities has been increasing drastically year by year, as seen in the following figure.

Figure 1.1 – Vulnerabilities trend over the past decades

Threat actors take advantage of such vulnerabilities and cause disruption to the confidentiality, integrity, or availability of the protected system. In certain vulnerabilities, the threat actor makes use of various exploits to deliver, install, and/or execute a malicious program on the system. Such malicious code is known as malware.

Malware comes in a variety of forms – viruses, worms, backdoors, trojans, adware, spyware, ransomware, and so on – each with its own characteristics. This malware aims to steal, damage, and/or destroy vulnerable systems – exfiltrating sensitive data or encrypting files and/or disks to make them unusable.

The damage caused by ransomware alone is shown in the following chart:

Figure 1.2 – Increasing cost of ransomware-related damage

Typical cyberattacks consist of a set of common phases or stages. Lockheed Martin has created a model called the Cyber Kill Chain to encapsulate these stages (https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html). The stages are as follows:

Reconnaissance: This is the phase in which the adversary identifies the target’s possible vulnerabilities and weak points. This may involve active scanning of the target network, passive information gathering, social engineering, gathering information from the internet and/or social networks, and so on. This step provides the adversary with sufficient information to proceed with the attack – such as which IP addresses are accessible, what ports are open, what applications are running, and details of the vulnerabilities on each.Weaponization: In this stage, the attacker creates a payload (weapon) that exploits the discovered vulnerability and plants malware on the victim’s machine.Delivery: This is the stage when the attacker delivers the prepared payload, for example, an infected document to the target. A typical delivery mechanism is a phishing email containing a malicious link or an infected PDF document.Exploitation: In this stage, the target machine is compromised by the exploit delivered in the previous stage. When the exploit code is executed, the attacker accomplishes their objective, such as remote control of the target machine. Subsequently, having gained a foothold on the victim’s machine, the attacker proceeds to the next phases, such as maintaining persistence and exfiltrating data.Installation: In the installation phase, various types of malwares are installed on the target machine – ransomware, backdoors, or trojans – based on the plan of the attacker for their purposes.Command and control: Once the malware is installed on the target machine, it typically contacts a command and control server. This may be to get additional instructions or commands to be executed on the target machine.Action: In this stage, the malware acts on the target as per the commands or instructions from the attacker. This may involve installing additional malware, exfiltrating sensitive data and system information back to the attacker-controlled server, or even performing denial-of-service attacks on any specified targets.

These are the typical stages of a cyberattack. From a security point of view, the earlier the attack is detected, the better. If the defense mechanisms in place can detect and stop an attack at the delivery stage, any compromise can be prevented.

In the next section, let us look at a strategy that aims to ensure the highest chance of a successful defense against attack attempts.

Defense-in-depth strategy

Defense in depth is a strategy for protecting a system against any attack using several independent defense methods. This approach was originally conceived by the National Security Agency. The system that needs to be protected consists of a set of resources and assets, including the network itself. A typical scenario would include web servers, mail servers, DNS infrastructure, WAN and LAN routers, authentication servers, database servers, laptops, and desktops.

As mentioned earlier, a defense-in-depth strategy uses independent and mutually exclusive mechanisms to protect and defend the assets; thus, the chances of detecting an attack are higher than using a single mechanism. It is sufficient for any one of the layers to detect the attack, in order to prevent and thwart it. The several layers of the defense-in-depth strategy are depicted in Figure 1.3.

Figure 1.3 – Defense in depth

The defense-in-depth strategy would include security technology, processes, and/or policies at several layers, including network, perimeter, endpoint, application, and data security.

Some of the various layers of the defense-in-depth approach in a typical scenario are discussed in the following subsections.

Firewalls (network and host layers)

Network firewalls filter the network by inspecting traffic that enters or leaves through network boundaries/zones. They enforce user-defined security policies across single or multiple network segments, comparing policies, adding threat modules, and assessing the data packets to prevent unauthorized access. Firewall deployments are precisely placed within the network to inspect and manage traffic flow.

Network firewalls are analogous to doorkeepers. When deployed in the network perimeter, they are typically the outermost layer in the defense-in-depth strategy. However, network firewalls are also deployed within a segregated network to separate various sections and/or departments. Network firewalls perform basic protocol decoding and analysis in order to be able to allow or deny packets and/or connections in or out of the network.

Host-based firewalls are like network firewalls except that they are concerned only with a single host as opposed to a set of hosts in a network.

Network- and host-based firewalls can create logs for every inbound and outbound connection that traverses through them. This can be immensely valuable from a detection point of view.

Intrusion detection and prevention systems (network and host layers)

IDS are analogous to security cameras. They are devices or programs that detect malicious activity against the concerned network or host (network-based or host-based IDS).

For a network-based IDS, the system inspects and analyzes the network traffic and tries to detect malicious activity based on signatures (for known attacks) or anomalous behavior or deviation from standard. The deviation from the standard can either be a statistical deviation (statistical anomaly-based IDS) or a deviation from protocol specifications (protocol anomaly-based IDS).

A host-based IDS will monitor all host artifacts in order to detect malicious activity, including network traffic to or from that host, process details, host-based logs, and files on the host.

IPS are IDS with the additional capability to enforce actions that prevent an attack. For example, upon detection of an attack, the IPS may drop the concerned packet or block the entire connection.

Endpoint detection and response (host layer)

Endpoint detection and response (EDR) comprises tools and technology that monitor activity on endpoint hosts and servers in order to detect malicious activity. The activity that is monitored by EDR includes processes, connections (to and from) the host, files created/modified, and registry changes.

Web application firewalls (network and host layers)

Web application firewalls (WAF) are firewalls specifically for web traffic. WAF inspect and analyze web traffic comprehensively. They can analyze both HTTP and HTTPS protocols. In the case of HTTPS, WAF often terminate the SSL sessions to decrypt the traffic, which often involves playing a man-in-the-middle role between the web client and the web server.

Traditional firewalls allow or deny traffic based on OSI layer 3 and 4 headers. Network-based IPS can perform limited application-level analysis. Compared to these, WAF are capable of comprehensive web (HTTP/HTTPS) traffic analysis in order to make the allow versus deny decision.

Some of the commercial companies that offer WAF are Fortinet, Barracuda, and Imperva. ModSecurity is also a widely available option for an open source WAF.

Mail security gateway (network)

A mail security gateway or firewall is another application-level firewall but for email-related protocols. A significant percentage of threats involve emails. In the first half of 2021, 75% of threats were delivered using email. Emails are often used as bait to trap unsuspecting users – by prompting them to open a malicious attachment, or by tempting them to click a malicious link.

Mail security gateways protect users from threats related to email by analyzing and filtering the malicious artifacts from an email. Mail firewalls perform deep inspection of the protocols related to mail, namely SMTP, POP, IMAP, and their encrypted counterparts.

Log management and monitoring (network and host)

Log management and monitoring solutions collect, inspect, and archive log messages and files from a variety of devices in the network. They also enable capabilities such as indexing and searching across the collected logs.

In the next section, let us specifically look at network IDS and IPS and the role that they play in the defense-in-depth strategy.

The role of network IDS and IPS

Network-based IDS and IPS play a significant role in the defense-in-depth strategy for information security. This role is unique when compared with other pieces of the defense-in-depth approach. As the name suggests, the primary role of IDS is detection, whereas IPS adds the extra capability of blocking the attack that it has detected.

The network IDS processes network traffic – analyzes the various protocols that are involved – with the goal of detecting malicious activity in a real-time fashion. The network IDS typically also has the capability to analyze packet captures offline; however, the most common case is to perform the analysis live so as to detect the attack in real time.

In general, the network IDS functionality would include the following:

Configuration management: IDS configuration essentially determines what exact functionality is performed by the IDS, how much memory needs to be allotted, the various parameters for learning for anomaly-based IDS, and the signatures to be analyzed.Packet acquisition module: This module is responsible for getting the network traffic data (packet data) from the source to the IDS. IDS often use packet capture libraries such as libpcap in order to attain this functionality.Decoder module: Irrespective of the type of IDS (signature-based or anomaly-based), there needs to be a module that can decode the various network protocols, maintain some state, and make the data available for the rest of the IDS to perform its detection operation.Detection module: This is the module that performs the detection functionality – whether it is signature matching or detecting an anomaly.Alert and log module: This module performs the task of generating an alert in the event of attack detection, as well as logging critical log messages regarding the IDS operation.

In the event of detecting an attack, the IDS/IPS generates an alert; these alerts are brought to the attention of a security operator for further action or sent to a central system such as Security Incident and Event Management (SIEM) for collection, correlation, and analysis. Figure 1.4 shows a typical IDS and IPS deployment scenario. It can be noted that the IPS is deployed in an inline fashion, whereas the IDS is deployed in an offline manner.

Figure 1.4 – Typical IDS and IPS deployment diagram

Due to the difference in their objectives, the IDS is typically deployed in a passive manner, often analyzing a copy of the network traffic (collected via a SPAN port on a router or firewall). IPS devices, on the other hand, operate in an inline mode – very similar to a firewall – so that they can block the offending packet or connection.

This difference – passive/offline versus inline – in the deployment leads to a key distinction. When the traffic rate increases to a level that the IDS cannot keep up with, it leads to packet drops; it does not affect the operation as it is a copy of the packet that was dropped. However, in the case of an inline operation, when the IPS cannot keep up with the rate of traffic leading to packet drops, it affects the network throughput and becomes a performance bottleneck. Therefore, there is increased demand on the IPS to have faster packet processing than for an IDS.

There is yet another key difference between the IDS and IPS, namely the consequence of a false positive. A false positive is when the IDS or IPS detects a benign packet or connection as malicious. For an IDS, this will result in a false positive alert being generated. This will result in an unnecessary alert and analysis. However, for an IPS that blocks packets and connections when an alert is generated, this will result in the interruption of a normal or benign connection, resulting in user dissatisfaction.

Due to these key differences, IDS and IPS devices are often configured very differently – one giving priority to detection (IDS) and the other giving priority to performance as well as detection (IPS).

In the next section, we will discuss how the IDS and IPS are categorized based on how the detectionis done.

Types of intrusion detection

Intrusion detection approaches are classified into the following based on how malicious activity is detected. The most common approaches are signature-based, anomaly-based, and hybrid. Let us discuss each of these approaches.

Signature-based intrusion detection

The signature-based approach uses predefined signatures in order to detect known threats. When an attack is initiated that matches one of these signatures, a predefined action (for example, generate an alert) is taken.

This is the most common approach for intrusion detection, especially in commercial solutions. Open source IDS/IPS – such as Snort and Suricata – are essentially signature-based. Signature-based systems are very good and proven to detect known attacks with very good accuracy and efficiency. As opposed to anomaly detection techniques, the signature-based IDS does not require any training or learning phase. The most important disadvantage of this approach is the inability to detect unknown attacks. Due to this reason, this approach requires constant (almost daily) updates to the signature set so that it can detect new threats that appear daily.

A simplified block diagram of a signature-based IDS is shown in Figure 1.5.

Figure 1.5 – Block diagram of a typical signature-based IDS

The input from the monitored environment (for example, packets from a monitored network) is processed and matched against a set of signatures; if there is a match, the system generates an alert. The quality of the system clearly depends on the quality of the signatures, and therefore maintaining and keeping the signatures updated is one of the main challenges of the system. The race between the attacker, who tries to create an exploit for a newly known vulnerability, and the defender (security operator), who attempts to create a signature that detects attacks against that vulnerability, is often a race against time.

Here is an example of an IDS (Snort) signature:

alert tcp any any -> $HOME_NET [80,8080] (msg:"SQL Injection Detected"; flow:established,to_server; http_uri; content:"/wordpress/wp-content/plugins/demo_vul/endpoint.php"; content:"union",distance 0; content:"select",distance 0,nocase; content:"from", distance 0; sid:123;)

This is a rule written to detect and alert on a SQL injection attempt to a web server operating on port 80 or 8080. An example would be the following:

http://acunetix.php.example/wordpress/wp-content/plugins/demo_vul/endpoint.php?user=-1+union+select+1,2,3,4,5,6,7,8,9,(SELECT+user_pass+FROM+wp_users+WHERE+ID=1)

The rule starts with the rule action, namely alert, which indicates the action that results if this rule matches. The subsequent terms indicate the protocol (tcp) that needs to be matched. The rule specifies the TCP destination ports of 80 and 8080. Typically, these will be HTTP traffic.

The msg keyword specifies the message to be included in the generated alert. The flow keyword specifies that this rule needs to be applied only to those TCP sessions that are in an ESTABLISHED state. Subsequently, the rule goes on to specify that the URI needs to contain certain specific strings.

This gives an idea and example of an IDS/IPS signature. The detailed understanding of such a signature is beyond the scope of this chapter and will be discussed in Chapter 14.

Anomaly-based intrusion detection

Anomaly-based intrusion detection detects malicious activity by how it differs from normal behavior. This often requires the system to define and/or learn normal behavior. Since the normal for one environment is often different than the normal for another environment, this approach typically requires a learning phase where the system learns the appropriate normal for a particular environment. During the learning phase, a baseline for normal activity is recorded; subsequently, in the running phase, the activity is compared against the baseline to detect anomalies.

One of the main advantages of this approach is that the anomaly-based approach does not require signatures, and the race against time for security coverage is not an issue. In other words, the anomaly-based approach can detect novel attacks that the IDS/IPS has not encountered before.

On the other hand, the main challenge for anomaly-based systems is that of false positives. Anomaly detection assumes that the outlier case is malicious. However, all outliers are not malicious, and this is the underlying reason for the high false positive rates associated with this approach. Subsequently, significant effort would be required to tune the system – to balance the false positives and false negatives.

Additionally, since the anomaly-based IDS generates alerts when there is a deviation from normal, the alert will not be specific; the system only knows that it is not normal. This results in non-specific or vague alerts being generated.

There are several sub-types of anomaly-based intrusion detection, namely the following:

Statistical anomaly-based: In the statistical anomaly-based approach, the IDS analyzes a set of predetermined values or variables (for example, packet sizes, login session variables, packet header values, and amount of data transferred) and maintains a baseline learned during the learning phase. Subsequently, the system analyzes the set of variables at runtime for deviation from the expected baseline. The system typically has a threshold setting that can be configured, and when the deviation from the predicted baseline is greater than the threshold, it detects the activity as malicious.Machine learning-based: Machine learning has made significant advances, and this approach is often used to detect outliers. Therefore, the technique is very good for anomaly detection-based IDS/IPS. This is a vast topic, but various techniques under machine learning can be used to detect unknown attacks.Protocol anomaly-based: This approach applies mainly to network-based IDS. Network traffic typically follows various network protocols. For example, email communication typically follows a set of protocols such as SMTP, IMAP, and POP. These protocols are clearly defined by specifications described in documents called RFC. Protocol anomaly-based IDS detect a deviation of network traffic from the concerned protocol’s RFC specification.

Anomaly detection can be a very powerful technique for detecting intrusions since it can detect new and unknown attacks, provided we can overcome the challenges, including high false-positive rates and tuning difficulties. One such technique combines anomaly detection with signature-based detection to create a hybrid solution.

Hybrid intrusion detection

As the name suggests, hybrid IDS combine signature-based and anomaly-based approaches to detect malicious activity. In the simplest design, the network traffic is processed by a signature-based component as well as an anomaly-based component, and the findings of each component are fed into a decision module that makes a final judgment on whether there is an attack or not.

In a more practical sense, typical IDS/IPS will be signature-based but may have some detection modules that work using an anomaly-based approach.

In the next section, let us discuss the state of the art in IDS/IPS. The section will discuss the important features present in the latest IDS/IPS.

The state of the art in IDS/IPS

The intrusion detection and prevention field has been evolving for a few decades. During this period, several commercial and open source IDS/IPS have been developed. As the nature of the internet and its protocols, as well as the complexity of threats, evolved, the IDS/IPS also had to evolve in order to keep up with the threats. Snort is an open source IDS/IPS that was created in 1998, and over the past 20+ years, it has evolved into one of the leading IDS/IPS software. Bro is another open source project, which started in 1994 and was mainly used in an academic setting for several years. Recently, it was renamed Zeek, and a community has formed around the open source project. Suricata is a relatively late player in the game and was created in 2009. It is a signature-based IDS/IPS similar to Snort. The rule syntax for Suricata is very similar to that of Snort. In addition to the rules, Suricata has many other similarities to Snort in functionality – although the design and implementation are completely different.

These three open source IDS/IPS have kept up with the challenges that they faced and stood the test of time. It may be said that the current state of these three IDS/IPS represents the state of the art in IDS/IPS. In this section, let us describe some of the challenges that these systems have faced and what features solved them.

Stateful analysis

Stateful analysis of the various network protocols is a necessary feature in any IDS/IPS. Snort was completely stateless and basically a packet analysis IDS in the initial years. Even when stateful analysis was introduced in the subsequent years, it was incomplete and insufficient. Ideally, the IDS/IPS must analyze the network traffic exactly as the end hosts would analyze it. This means that the IDS would need to maintain a very similar state to the end hosts. This is not a trivial task. This is the reason why it took decades for Snort to improve its stateful analysis functionality. Currently, one could say that Snort is a stateful IDS/IPS device, even though there are still limitations.

Fast packet acquisition

Historically, IDS/IPS devices used the packet capture library called libpcap. This is a library used by the tcpdump project and was available as open source. libpcap worked great, but as the internet speed increased, this library started becoming a performance bottleneck. In the case of libpcap, the packet data (network traffic) had to be copied several times before reaching the IDS for processing, and this was one of the reasons for the performance issue. Currently, the state of the art uses zero-copy mechanisms in order to improve performance. Although Snort still supports and offers libpcap-based packet acquisition, it offers all the latest packet acquisition mechanisms to be used.

Parallel processing

The state of the art is for the IDS/IPS to perform network traffic analysis using parallel processing – this could be a multi-process-based or multi-thread-based design. Snort started as a single-threaded, single-process IDS, and then evolved into a multi-process design. Currently, Snort uses a multi-threaded design.

In a multi-process and multi-threaded design, an incoming session would be processed by one of the processes or threads. Once a session is analyzed by a process or thread, then all the subsequent network packets for that session will be analyzed by that process or thread. This is called session pinning. Typically, such pinning is based on a hashing approach, where the hash will be based on the source and destination IP addresses, port, and protocol. However, in this approach, two related sessions that hash to two separate processes or threads will result in a lesser-grade analysis.

Pattern matching

Pattern matching has been and is still one of the most important features of IDS/IPS. A single signature may contain several pattern matches. Originally, these were evaluated one rule at a time, one pattern at a time. With time, multi-pattern search algorithms were used in order to speed up the rule processing.

In addition, as opposed to the crude pattern matching of the past, current IDS/IPS devices perform the pattern matching with context. For example, when a pattern is specified, it can also be specified what data to match against – HTTP URI, HTTP header, and so on. This improves the performance since the pattern search can be limited to specific data, and it also improves detection accuracy.

Extending rule language

Most IDS/IPS have a rich rule language. However, there will always be cases that cannot be covered by the limited capability offered by the rule language. Each system – Snort, Suricata, and Zeek – has its own approach to this challenge. Zeek from the Bro days had a full-fledged language to write detections in. So, the challenge really did not apply to Zeek. Snort came up with shared object (SO) rules, whereby custom C code could be written for a particular functionality and released as .so files in a release. Suricata integrated Lua scripting as part of the rule language extension.

App and protocol identification

Historically, Snort rules were based on protocol and port. For example, the rule would specify that it applies to TCP and on ports 80, 8080, and 3128. The list of ports could be more extensive to cover the usual HTTP ports. However, if there is an HTTP session on port 1000, the rule will not be applied against that session. This challenge was solved by introducing the app and protocol identification feature, which is a state-of-the-art feature. All leading IDS/IPS detect the various protocols on any random port to perform the analysis correctly.

File analysis

In certain cases, the