AI and Machine Learning for Network and Security Management E-Book

Yulei Wu

• Herausgeber: John Wiley & Sons
• Kategorie: Wissenschaft und neue Technologien
• Serie: IEEE Press Series on Network Management
• Sprache: Englisch

Seitenzahl: 456

Veröffentlichungsjahr: 2022

Beschreibung

AI AND MACHINE LEARNING FOR NETWORK AND SECURITY MANAGEMENT

Extensive Resource for Understanding Key Tasks of Network and Security Management

AI and Machine Learning for Network and Security Management covers a range of key topics of network automation for network and security management, including resource allocation and scheduling, network planning and routing, encrypted traffic classification, anomaly detection, and security operations. In addition, the authors introduce their large-scale intelligent network management and operation system and elaborate on how the aforementioned areas can be integrated into this system, plus how the network service can benefit.

Sample ideas covered in this thought-provoking work include:

• How cognitive means, e.g., knowledge transfer, can help with network and security management
• How different advanced AI and machine learning techniques can be useful and helpful to facilitate network automation
• How the introduced techniques can be applied to many other related network and security management tasks

Network engineers, content service providers, and cybersecurity service providers can use AI and Machine Learning for Network and Security Management to make better and more informed decisions in their areas of specialization. Students in a variety of related study programs will also derive value from the work by gaining a base understanding of historical foundational knowledge and seeing the key recent developments that have been made in the field.

Leseprobe

Table of Contents

Cover

Title Page

Copyright

Author Biographies

Preface

Acknowledgments

Acronyms

1 Introduction

1.1 Introduction

1.2 Organization of the Book

1.3 Conclusion

References

2 When Network and Security Management Meets AI and Machine Learning

2.1 Introduction

2.2 Architecture of Machine Learning‐Empowered Network and Security Management

2.3 Supervised Learning

2.4 Semisupervised and Unsupervised Learning

2.5 Reinforcement Learning

2.6 Industry Products on Network and Security Management

2.7 Standards on Network and Security Management

2.8 Projects on Network and Security Management

2.9 Proof‐of‐Concepts on Network and Security Management

2.10 Conclusion

References

Notes

3 Learning Network Intents for Autonomous Network Management*

3.1 Introduction

3.2 Motivation

3.3 The Hierarchical Representation and Learning Framework for Intention Symbols Inference

3.4 Experiments

3.5 Conclusion

References

Notes

4 Virtual Network Embedding via Hierarchical Reinforcement Learning

1

4.1 Introduction

4.2 Motivation

4.3 Preliminaries and Notations

4.4 The Framework of VNE‐HRL

4.5 Case Study

4.6 Related Work

4.7 Conclusion

References

Note

5 Concept Drift Detection for Network Traffic Classification

5.1 Related Concepts of Machine Learning in Data Stream Processing

5.2 Using an Active Approach to Solve Concept Drift in the Intrusion Detection Field

5.3 Concept Drift Detector Based on CVAE

5.4 Deployment and Experiment in Real Networks

5.5 Future Research Challenges and Open Issues

5.6 Conclusion

References

Note

6 Online Encrypted Traffic Classification Based on Lightweight Neural Networks

*

6.1 Introduction

6.2 Motivation

6.3 Preliminaries

6.4 The Proposed Lightweight Model

6.5 Case Study

6.6 Related Work

6.7 Conclusion

References

Notes

7 Context‐Aware Learning for Robust Anomaly Detection

*

7.1 Introduction

7.2 Pronouns

7.3 The Proposed Method – AllRobust

7.4 Experiments

7.5 Discussion

7.6 Conclusion

References

Note

8 Anomaly Classification with Unknown, Imbalanced and Few Labeled Log Data

8.1 Introduction

8.2 Examples

8.3 Methodology

8.4 Experimental Results and Analysis

8.5 Discussion

8.6 Conclusion

References

Notes

9 Zero Trust Networks

9.1 Introduction to Zero‐Trust Networks

9.2 Zero‐Trust Network Solutions

9.3 Machine Learning Powered Zero Trust Networks

9.4 Conclusion

References

10 Intelligent Network Management and Operation Systems

10.1 Introduction

10.2 Traditional Operation and Maintenance Systems

10.3 Security Operation and Maintenance

10.4 AIOps

10.5 Machine Learning‐Based Network Security Monitoring and Management Systems

10.6 Conclusion

References

11 Conclusions, and Research Challenges and Open Issues

11.1 Conclusions

11.2 Research Challenges and Open Issues

References

Index

End User License Agreement

List of Tables

Chapter 2

Table 2.1 Examples of supervised ML applications.

Table 2.2 Examples of semi‐ and unsupervised ML applications.

Table 2.3 Examples of policy‐based RL applications.

Table 2.4 Examples of value‐based RL applications.

Table 2.5 Summary of network management products.

Table 2.6 Summary of security management products.

Table 2.7 Standards on network management.

Table 2.8 ISG ENI's specifications on Cognitive Network Management.

Table 2.9 Standards on security management.

Table 2.10 Projects on network and security management using ML techniques....

Chapter 5

Table 5.1 Database description.

Table 5.2 Feature selection.

Chapter 6

Table 6.1 The details of Dataset A.

Table 6.2 A summary of existing traffic classification methods.

Chapter 7

Table 7.1 Label correspondence of BGL dataset.

Table 7.2 Label correspondence of Thunderbird dataset.

Table 7.3 Software and hardware environment for supervised learning experim...

Table 7.4 The number of logs in each category of the BGL dataset after filt...

Table 7.5 Input parameters of a translation API.

Table 7.6 The experimental results obtained from models after training with...

Chapter 8

Table 8.1 The details of datasets.

Chapter 10

Table 10.1 Open‐source operation and maintenance system.

Table 10.2 Open‐source tools related to access control.

Table 10.3 Security audit and intrusion detection‐related tools.

Table 10.4 Penetration testing‐related tools.

Table 10.5 Vulnerability detection related tools.

Table 10.6 CI/CD related tools.

Table 10.7 Honeypot‐related tools.

Table 10.8 Data security‐related tools.

Table 10.9 Open‐source AIOps.

Table 10.10 Open‐source AIOps algorithms and packages.

List of Illustrations

Chapter 1

Figure 1.1 The chapter organization of this book.

Chapter 2

Figure 2.1 The overall architecture of AI‐ and ML‐empowered network and secu...

Figure 2.2 The framework of reinforcement learning.

Figure 2.3 The description of the VNE process.

Figure 2.4 The framework of MCTS.

Figure 2.5 The processing flow of Poseidon.

Figure 2.6 The processing flow of NetworkML.

Figure 2.7 The processing flow of Credential‐Digger.

Figure 2.8 The processing flow of classification.

Figure 2.9 The processing flow of active learning.

Figure 2.10 The processing flow to overcome concept drift.

Chapter 3

Figure 3.1 The semantic triangle.

Figure 3.2 The triangle theory of intention symbol semantics.

Figure 3.3 Examples of intention symbol semantics.

Figure 3.4 Connectivity intention symbol.

Figure 3.5 Deadlock free intention symbol.

Figure 3.6 Whole path intention symbol.

Figure 3.7 “Paths” satisfying the shortest intention.

Figure 3.8 Paths satisfying the connectivity intention.

Figure 3.9 Three priority modes of intention symbols.

Figure 3.10 The probability distribution

under the condition that

takes ...

Figure 3.11 The probability distribution

under the condition that

takes ...

Figure 3.12 The probability distribution

.

Figure 3.13 The probability distribution of three structure modes.

Figure 3.14 The probability distribution of symbol structures.

Figure 3.15 IBM network topology.

Figure 3.16 CWIX network topology.

Figure 3.17 BT Europe network topology.

Figure 3.18 Darkstrand network topology.

Chapter 4

Figure 4.1 IaaS business model.

Figure 4.2 A typical scene of a VNE problem. The edge weights and node weigh...

Figure 4.3 The architecture of the VNE‐HRL.

Figure 4.4 The algorithm performance over time on the test dataset. (a) The ...

Figure 4.5 The test results of different upper bounds of resource requests o...

Chapter 5

Figure 5.1 Main processes of the active approach‐addressing concept drift.

Figure 5.2 Overview of the semisupervised methodology.

Figure 5.3 Results of the active approach to solve concept drift.

Figure 5.4 The structure of CVAE.

Figure 5.5 TDR and FAR in two datasets. (a) TDR in SINE, (b) TDR in SINE_g, ...

Figure 5.6 The network topology in experiments.

Figure 5.7 The improvement of various schemes of the classifier on real‐worl...

Chapter 6

Figure 6.1 The preprocessing and architecture of LightNet. The structure of ...

Figure 6.2 The construct of attention encoder.

Figure 6.3 Dataset A: comparisons between LightNet and the three baseline mo...

Figure 6.4 Dataset B: comparisons between LightNet and the three baseline mo...

Figure 6.5 Dataset B: comparisons with the three baseline models. The values...

Chapter 7

Figure 7.1 The example of evolution of a log.

Figure 7.2 The principle of FixMatch.

Figure 7.3 An example of logs.

Figure 7.4 The process of log anomaly detection.

Figure 7.5 A Darin parse tree with a depth of 3.

Figure 7.6 The original parse tree.

Figure 7.7 The updated parse tree.

Figure 7.8 The structure of FastText.

Figure 7.9 The generation process of ACU.

Figure 7.10 The generation process of

.

Figure 7.11 The structure of attention‐based Bi‐LSTM.

Figure 7.12 An example of HDFS logs.

Figure 7.13 An example of BGL logs.

Figure 7.14 An example of Thunderbird logs.

Figure 7.15 Experimental results of LogRobust, Consen–LogRobust, and AllRobu...

Figure 7.16 The accuracy achieved by models after training on imbalanced HDF...

Figure 7.17 Experimental results of LogRobust, SMOTE‐LogRobust, and AllRobus...

Figure 7.18 The workflow of back‐translation.

Figure 7.19 The data distribution of BGL‐Raw training dataset.

Figure 7.20 The data distribution of BGL‐Imbalanced training dataset.

Figure 7.21 The experimental results of models after training with only 6

l...

Figure 7.22 The recall achieved after training with 24

labels on HDFS train...

Chapter 8

Figure 8.1 The components of a Log.

Figure 8.2 Logging and log collection.

Figure 8.3 The process of creating event count matrices.

Figure 8.4 Comparison of learning with sufficient and few training samples. ...

Figure 8.5 Different strategies on how few‐shot learning methods can solve t...

Figure 8.6 Log parsing tree‐using Drain with depth of 3.

Figure 8.7 The architecture of OpenLog.

Figure 8.8 The encoder module. (a) Word‐level encounter and (b) Sentence‐lev...

Figure 8.9 The prototypical module.

Figure 8.10 The relation module.

Figure 8.11 The baselines. (a) LogRobust and (b) LogClass.

Figure 8.12 The examples of BGL, Thunderbird, Liberty, and Spirit logs.

Figure 8.13 The samples proportion of raw imbalanced log in Thunderbird, Lib...

Figure 8.14 A comparison of the evaluation scores against the baselines on T...

Figure 8.15 A comparison of the evaluation scores against the baselines on T...

Figure 8.16 A comparison of the evaluation scores against the baselines on T...

Chapter 9

Figure 9.1 The core logical components of zero‐trust networks.

Figure 9.2 The components and access flows of BeyondCorp.

Figure 9.3 The access flow of SDP.

Figure 9.4 Micro‐segmentation information in the VXLAN packet header.

Figure 9.5 The VXLAN network example.

Figure 9.6 The trust algorithm process in zero‐trust networks.

Figure 9.7 The authorization system in zero‐trust networks.

Figure 9.8 Feature fusion framework proposed by Han et al. (2020).

Figure 9.9 Feature Fusion Framework Proposed by Guo et al. (2021).

Chapter 10

Figure 10.1 The Nagios structure.

Figure 10.2 The Zabbix structure.

Figure 10.3 The Prometheus structure (https://prometheus.io/docs/introductio...

Figure 10.4 The metis code structure.

Figure 10.5 metis time series detector.

Figure 10.6 UAVStack HM structure.

Figure 10.7 Skyline simplified workflow.

Figure 10.8 Machine learning‐based network security monitoring and managemen...

Guide

Cover

Table of Contents

Title Page

Copyright

Author Biographies

Preface

Acknowledgments

Acronyms

Begin Reading

Index

End User License Agreement

Pages

ii

iii

iv

xiii

xv

xvi

xvii

xix

xx

xxi

xxii

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

165

166

167

168

169

170

171

172

173

174

175

176

177

179

180

181

182

183

184

185

187

188

189

190

191

192

193

195

196

197

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

IEEE Press

445 Hoes Lane Piscataway, NJ 08854

IEEE Press Editorial Board

Sarah Spurgeon,

Editor in Chief

Jón Atli Benediktsson

Andreas Molisch

Diomidis Spinellis

Anjan Bose

Saeid Nahavandi

Ahmet Murat Tekalp

Adam Drobot

Jeffrey Reed

Peter (Yong) Lia

Thomas Robertazzi

AI and Machine Learning for Network and Security Management

Yulei WuUniversity of ExeterUK

Jingguo GeIIE Chinese Academy of SciencesChina

Tong LiIIE Chinese Academy of SciencesChina

Copyright © 2023 by The Institute of Electrical and Electronics Engineers, Inc. All rights reserved.

Published by John Wiley & Sons, Inc., Hoboken, New Jersey.Published simultaneously in Canada.

No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per‐copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750‐8400, fax (978) 750‐4470, or on the web at www.copyright.com. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748‐6011, fax (201) 748‐6008, or online at http://www.wiley.com/go/permission.

Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose. No warranty may be created or extended by sales representatives or written sales materials. The advice and strategies contained herein may not be suitable for your situation. You should consult with a professional where appropriate. Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages. Further, readers should be aware that websites listed in this work may have changed or disappeared between when this work was written and when it is read. Neither the publisher nor authors shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages.

For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762‐2974, outside the United States at (317) 572‐3993 or fax (317) 572‐4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic formats. For more information about Wiley products, visit our web site at www.wiley.com.

Library of Congress Cataloging‐in‐Publication Data applied for:Hardback ISBN: 9781119835875

Cover Design: WileyCover Image: © Bill Donnelley

Author Biographies

Yulei Wu, is a Senior Lecturer with the Department of Computer Science, Faculty of Environment, Science and Economy, University of Exeter, UK. His research focuses on networking, Internet of Things, edge intelligence, information security, and ethical AI. He serves as an Associate Editor for IEEE Transactions on Network and Service Management, and IEEE Transactions on Network Science and Engineering, as well as an Editorial Board Member of Computer Networks, Future Generation Computer Systems, and Nature Scientific Reports at Nature Portfolio. He is a Senior Member of the IEEE and the ACM, and a Fellow of the HEA (Higher Education Academy).

Jingguo Ge, is currently a Professor of the Institute of Information Engineering, Chinese Academy of Sciences (CAS), and also a Professor of School of Cyber Security, University of Chinese Academy of Sciences. His research focuses on Future Network Architecture, 5G/6G, Software‐defined networking (SDN), Cloud Native networking, Zero Trust Architecture. He has published more than 60 research papers and is the holder of 28 patents. He participated in the formulation of 3 ITU standards on IMT2020.

Tong Li, is currently a Senior Engineer of Institute of Information Engineering at the Chinese Academy of Sciences (CAS). His research and engineering focus on Computer Networks, Cloud Computing, Software‐Defined Networking (SDN), and Distributed Network and Security Management. He participated 2 ITU standards on IMT2020 and developed many large‐scale software systems on SDN, network management and orchestration.

Preface

With the fast development of networking technologies, the communication network has gone through four generations and is in the process of deploying the fifth‐generation system (5G) worldwide. 5G has its unique feature of accommodating diversified services on top of a shared infrastructure. These services not only include the telecommunication service that we use every day for our daily lives, but also encompass a wide variety of services in support of many important vertical industries including energy, health, water, manufacturing, environment, to name a few. These services are mainly classified into three broad categories: enhanced Mobile Broadband (eMBB), Ultra Reliable Low Latency Communications (URLLC), and massive Machine Type Communications (mMTC). The deployment of 5G to support eMBB services has already started in the globe, and that of supporting URLLC and mMTC will start in the foreseeable future. Meanwhile, research of next‐generation communication systems, i.e. beyond 5G (B5G) or 6G, has already started with many research centers and groups established globally.

To meet requirements of diverse services running on top of 5G/B5G/6G infrastructure, many networking and computing techniques have been incorporated into the communication system, including reconfigurable intelligent surface, Millimeter‐wave/THz links, high‐capacity backhaul connectivity, cloud nativeness, machine‐type communications, edge intelligence, blockchain, and quantum computing. This immediately results in the increasing complexity of modern networking and communication systems which will be at a scale and scope we have never seen before. This also remarkably raises the bar for network and service management. Since network security is an integral part of network management, the broad understanding of network management shall cover both network management and security management. The closed‐form models created for individual protocols, applications, and systems have been successful for network and security management in the past 20 years. However, the networking systems of today are too complicated for closed‐form analysis.

Network automation is being pursued by the community to facilitate network and security management. Artificial intelligence (AI) and machine learning (especially deep learning) have been widely used in a number of fields, e.g. image recognition and computer vision as well as natural language processing, to enhance the automation of relevant tasks. The networking community started to adopt AI and machine learning techniques to achieve the goal of network automation in recent years. Due to the complexity of today's networking systems, it is essentially hard to make a fully autonomous system. The current start‐of‐the‐art shows a promising progress of developing AI and machine learning models to automate certain tasks for network and security management, including network planning and routing, resource allocation and scheduling, encrypted traffic classification, anomaly detection, zero trust networks, and security operations.

This book covers the key tasks of network and security management, and elaborates how advanced AI and machine learning techniques can improve the network automation. It will not only address the problems from the computing point of view, but also explore how the cognitive means, e.g. knowledge transfer, can help with the network and security management. Network automation has become a burning issue for network and security management. This book will be useful and helpful for network engineers to tackle network automation issues, and it will also be a good textbook for education in universities. The book can also help policy‐makers understand how network automation works in the field of network and security management.

Yulei Wu    

Exeter, UK    

Jingguo Ge  

Beijing, China

Tong Li     

Beijing, China

Acknowledgments

Many thanks to all the contributors of this book, including Guozhi Lin, Lei Zhang, Zhaoxue Jiang, Peijie Sun, Jin Cheng, Zhenguo Zhang, and Zhibin Xu.

Acronyms

1D‐CNN

one‐dimension convolutional network

2D‐CNN

two‐dimension convolutional network

5G

fifth‐generation mobile communication system

ABAC

attribute‐based access control

ACL

access control list

ACU

adaptive context unit

ADASYN

oversample using adaptive synthetic

AE

AutoEncoder

AH

accepting host

AI

artificial intelligence

AIOps

artificial intelligence for it operations

ANN

artificial neural network

AP

access proxy

ART

adversarial robustness tool

Bi‐GRU

bidirectional gate recurrent unit

Bi‐LSTM

bidirectional long‐short term memory

CAPEX

capital expenditure

CART

classification and regression tree

CNN

convolutional neural network

CPU

central processing unit

CSTNET

China Science and Technology Network

CVAE

conditional variational autoencoder

DAE

denoising AutoEncoder

DBSCAN

density‐based spatial clustering of applications with noise

DDM

drift detection method

DDQN

double deep Q network

DNS

domain name system

E2E

end‐to‐end

EDR

endpoint detection and response

EM

expectation maximum

EPG

end point group

FAR

false alarm rate

FN

false negative

FP

false positive

GAN

generative adversarial network

GBP

group‐based police

GCN

graph convolutional network

GDPR

general data protection regulation

GFE

Google front end

GPU

graphics processing unit

HEA

high‐level VNE agent

HMM

hidden Markov model

HRL

hierarchical reinforcement learning

HTTP

hypertext transfer protocol

HTTPS

hypertext transfer protocol over secure socket layer

ICT

information and communication technology

IDS

intrusion detection system

IH

initiating host

IIoT

Industrial Internet of Things

ILP

integer linear programming

InP

infrastructure provider

IoT

Internet of Things

ISE

identity services engine

ISP

internet service provider

KNN

K‐nearest neighbor

KPI

key performance indicator

LEA

low‐level VNE agent

LSTM

long short‐term memory

MCTS

Monte Carlo tree search

MDP

Markov decision process

MEC

mobile edge computing

MILP

mixed‐integer linear programming

ML

machine learning

MRE

machine reasoning engine

MSE

mean square error

NFV

network functions virtualization

NGMN

next generation mobile network

O&M

operation and maintenance

ONF

open networking foundation

OPEX

operating expenditure

PA

policy administrator

PCA

principal component analysis

PE

policy engine

PEP

policy enforcement point

PKI

public key infrastructure

PSO

particle swarm optimization

QoS

quality of service

QUIC

quick UDP Internet connection

RBM

restricted Boltzmann machine

ResNet

residual network

RL

reinforcement learning

RNN

recurrent neural network

SAE

sparse AutoEncoder

SDN

software‐defined networking

SDP

software‐defined perimeter

SD‐RAN

software‐defined radio access network

SD‐WAN

software‐defined wide area network

seq2seq

sequence to sequence

SIEM

security information and event management

SLA

service‐level agreement

SMDP

semi‐Markov decision process

SMOTE

synthetic minority over‐sampling technique

SMTP

simple mail transfer protocol

SN

substrate network

SN

substrate network

SNI

service name indication

SP

service provider

SSI

symbolic structure inferring

SSL

symbolic semantic learning

SSO

single sign‐on

SVM

support vector machine

TA

trust algorithm

TCP

transmission control protocol

TDG

traffic dispersion graph

TDR

true detection rate

TP

true positive

VAE

variational autoencoder

VLAN

virtual local area network

VN

virtual network

VNE

virtual network embedding

VNR

virtual network request

VoIP

voice over internet protocol

VPN

virtual private network

VR

virtual reality

VXLAN

virtual extensible local area network

ZTN

zero trust network

1Introduction

1.1 Introduction

Networking systems have been experiencing rapid advancement in recent years, due to the fast development of 5G (Cheng et al., 2018, 2020b, Wu et al., 2021b), Internet of Things (IoT) (Wu et al., 2021c, Wu, 2021), Cloud/Edge Computing (Zhang et al., 2020, Wu, 2020), and Industry 4.0 (Wu et al., 2021a, Turner et al., 2021). On the one hand, many advanced networking techniques have been developed, such as software‐defined networking (SDN) (Miao et al., 2016, Wang et al., 2018, Yang et al., 2020), network functions virtualization (NFV) (Miao et al., 2019, Cheng et al., 2020b), and network slicing (Wang et al., 2019, 2020) to facilitate network and service deployment and management. On the other hand, cybersecurity is a major concern for networking systems due to the increase in system exposure to the Internet (Wu et al., 2021a, Garg et al., 2020, Culot et al., 2019). Many security mechanisms, e.g. intrusion detection, traffic classification, and anomaly detection, have been developed to facilitate the security management of networking systems (Huang et al., 2017, 2018, Zuo et al., 2020, Sun et al., 2020).

Telecommunication networks such as 5G have received significant attention in the past few years because of their capabilities of accommodating diverse vertical industry applications (Wang et al., 2019, 2020). Along with the diversified services as well as their changing and/or stringent service requirements, 5G networks have become a complex system that requires advanced artificial intelligence (AI) and machine‐learning (ML) techniques to manage and maintain high‐standard services to users (Yan et al., 2020). From the perspective of network operators, it is important to maximize the resource utilization of 5G infrastructure, while minimizing the violation of service‐level agreement (SLA) (Wang et al., 2019). The research of next‐generation telecommunication networks, the so‐called 6G (Wu et al., 2021d), has been initiated by many countries, such as United Kingdom, USA, China, Finland, just to name a few. “AI Everywhere” is an important component for 6G to ensure an automatic, healthy, and secure networking system.

The fast advancement of IoT and Industrial Internet of Things (IIoT) is transforming many traditional industries (many of them are critical infrastructures), such as energy, healthcare, factory, and transportation, toward the goal of Industry 4.0 (Wu et al., 2021a). Such a complex networking system, connecting tens of billions of devices to the Internet, is collecting a huge amount of data every day. AI and ML techniques can leverage the knowledge learned from the data to automate many tasks for these industries (Lin et al., 2021), resulting in the so‐called “smart energy, smart factory, smart transportation,” to name a few. Such an automation remarkably increases the efficiency of system operation of industries. However, since traditional form of these industries is much more isolated, the exposure of these industries to the Internet as a result of the transformation, calls for significant security management to ensure the safety of these critical infrastructures (Culot et al., 2019, Wu et al., 2021a).

In order to properly apply AI and ML technologies into the field of network and security management, many real‐world conditions and challenges need to be considered. For example, network intent is a key piece of information to enable autonomous network management (Lin et al., 2021). How to gain accurate network intent from network big data and how to ensure that the learned network intent can be readily used across different network environments is nontrivial. Reinforcement learning (RL) is a useful tool for autonomous network management (Yan et al., 2020). Successfully applying RL in various network management tasks is challenging. In many real‐world conditions, such as IoT/IIoT, lightweight learning models are required (Cheng et al., 2020a). How to devise such models while maintaining the model performance is still worth to investigate for the field of network and security management. In addition, learning from encrypted data, e.g. encrypted traffic, is crucial, due to the increase in the volume of such traffic enforced by data regulations like the general data protection regulation (GDPR) (Liu et al., 2020). Further, because of the changing condition of real‐world networking systems, network data are not ideal in many cases. They are usually evolving, changing, and imbalanced, and new data that have not been seen before may present from time to time. Besides, network data are usually hard to label, resulting in few‐shot issues. How to effectively learn useful information from such “noisy” data is of paramount importance to ensure the success of AI‐enabled network and security management (Sun et al., 2020).

In this book, we provide our insights and potential solutions to the above issues and challenges and consider various applications to network and security management including autonomous networks, resource allocation, traffic processing, traffic classification, anomaly detection, anomaly classification, and zero trust networks (ZTNs). In Section 1.2, we will explain the rationale under which the chapters in this book are organized.

1.2 Organization of the Book

There are two strands in this book. The first strand is in Chapter 2, where we provide a comprehensive review of potential AI and ML techniques for network and security management, the existing industry products, standards, projects, and proof‐of‐concepts. The second strand is across Chapters 3–9, where we elaborate the application of AI and ML techniques in various network and security management tasks. In Chapter 10, we elaborate an intelligent network management and operation system and discuss the deployment of the proposed solutions in this book. In Chapter 11, we conclude this book and provide potential research challenges and open issues that will be useful for future research in this area. Figure 1.1 shows the chapter organization of this book. In what follows, we briefly introduce each chapter to facilitate readers understand the content of this book.

Chapter 2

. This chapter discusses the status and limitations of current network and security management and proposes an architecture for ML‐empowered network and security management. Well‐known AI and ML techniques that are useful for network and security management are reviewed and discussed. We also investigate existing industry products, standards, and proof‐of‐concepts for network and security management.

Chapter 3

. The realization of network autonomy requires network knowledge to manage the network. The abstract intent of network management tasks can be considered as part of network knowledge. In this chapter, we treat abstract intents of network management tasks as a composite structure of symbols. Each symbol expresses the intention of the network management task in a certain aspect. The combinations of symbols, representing a network management task, should be able to be transferred and implemented across different networks. In this regard, we design a reference mechanism for learning intention symbols and their structures from network data. Taking path selection as an example, we describe in detail how to implement this mechanism to obtain the intent structure of the path selection task. It has been proved by experiments that the knowledge learnt by the proposed solution can be transferred and effectively leveraged in different network environments.

Chapter 4

. Due to the outstanding performance of automatic exploration and quick development, RL methods have been applied to the

virtual network embedding

(

VNE

) problem. In this chapter, we find that a proactive VNE algorithm can benefit from

hierarchical reinforcement learning

(

HRL

). In this algorithm, a two‐level agent is responsible for executing the VNE task, considering both the long‐term impact and short‐term impact. At the high level, the agent selects a feasible request from a batch, which aims to maximize the long‐term revenue. At the low level, the agent manages to embed the selected request with the minimum cost.

Chapter 5

. Although network traffic classification algorithms based on machine learning can alleviate the limitations imposed by traditional techniques, most of them are carried out by learning an underlying concept (i.e. data distribution) from a static dataset. Due to the exponential increase in the available network data, considerable attention has been received on processing network data as a stream. In this scenario, due to unforeseen circumstances in the network, the phenomenon of concept drift will degrade the performance of the classifier. In this chapter, after measuring the impact of concept drift on network traffic classifiers, we present a concept drift detector based on

conditional variational autoencoder

s (

CVAE

s) under the semisupervised learning. In addition, we deploy the detector in a real‐world environment, and experimental results show that this algorithm plays a great role in stabilizing the performance of a classifier.

Chapter 6

. The surge in the volume of encrypted traffic and the nontransparency of encrypted traffic leads to high computational overheads in efficient network management. In this chapter, we introduce a lightweight and online approach for traffic classification, which adopts the multihead attention mechanism and the convolutional networks. Due to the one‐step interaction of all packets and the parallel computing, the multihead attention mechanism can significantly reduce the number of model parameters and the running time. In addition, the effectiveness and efficiency of convolutional networks are proved in traffic classification.

Chapter 7

. As the scale of networking systems expands, a fast‐growing number of logs are produced. This chapter proposes a robust context‐aware method for log anomaly detection. It combines word embedding with region embedding to conduct log vectorization. Such rich semantic information enables the proposed method to deal with unseen log data and understand imbalanced log data better and deeper. The proposed method combines semisupervised learning to make full use of labeled data and unlabeled data.

Chapter 8

. ML‐based log anomaly classification methods have been widely studied to ensure the stability and reliability of large‐scale systems. This chapter briefly introduces the feature extraction in log analysis and the few‐shot problem by examples. Then, we propose OpenLog, an anomaly classification method based on meta‐learning. OpenLog uses a two‐layer semantic encoder to simplify the complex feature engineering. It adopts the meta‐learning strategy to train the models using sufficient auxiliary datasets to enhance its performance. OpenLog transforms the multiclassification task into a binary‐classification task, and it can classify unseen anomalies without retraining.

Chapter 9

. In recent years, many

advanced persistent attack

s (

APT

s) have occurred on corporate internal networks. Traditional perimeter‐based security defense techniques such as firewalls, which assume that users and devices inside a network are safe and trustworthy, can no longer provide sufficient security protection. The concept of ZTN was therefore proposed. In ZTN, every request, whether it comes from an internal network or an external network, must be authenticated and authorized before accessing resources. In this chapter, we provide a brief introduction of ZTN, including its concept, its architecture, and its current implementation schemes such as access proxy‐based,

software‐defined perimeter

(

SDP

)‐based, microsegmentation‐based solutions, to name a few. Since ZTN needs to authenticate and authorize requests, it is necessary to consider as many devices, users, and environmental information as possible to make decisions. As there are a large number of services, traffic, and equipment logs in the corporate intranet, ML‐based information fusion and decision‐making methods may improve authentication and authorization performance. Therefore, in this chapter, we evaluate the possibility of using ML in ZTN.

Chapter 10

. Although various intelligent operation and management technologies based on deep learning are being developed, how to efficiently apply them to real‐world products is one of the core challenges faced by deep learning. In this chapter, we introduce various open source tools, frameworks, and characteristics in the field of operations management and security. Furthermore, we analyze existing security operations and management systems based on deep learning. Finally, we propose a security framework for intelligent operation and management based on network big data and describe the core functions and interfaces in the framework.

Chapter 11

. This chapter provides a brief summary of this book, followed by a list of important research challenges and open issues that can be used for further research on AI and ML for network and security management.

Figure 1.1 The chapter organization of this book.

1.3 Conclusion

This chapter provided a brief introduction of this book, emphasizing the motivation of writing this book and the chapter organization of the book. In addition, a brief review of each chapter is also provided, facilitating readers understand the content of this book.

References

Xiangle Cheng, Yulei Wu, Geyong Min, and Albert Y. Zomaya. Network function virtualization in dynamic networks: A stochastic perspective.

IEEE Journal on Selected Areas in Communications

, 36(10):2218–2232, 2018. doi: 10.1109/JSAC.2018.2869958.

Jin Cheng, Runkang He, E Yuepeng, Yulei Wu, Junling You, and Tong Li. Real‐time encrypted traffic classification via lightweight neural networks. In

GLOBECOM 2020 ‐ 2020 IEEE Global Communications Conference

, pages 1–6, 2020a. doi: 10.1109/GLOBECOM42002.2020.9322309.

Xiangle Cheng, Yulei Wu, Geyong Min, Albert Y. Zomaya, and Xuming Fang. Safeguard network slicing in 5G: A learning augmented optimization approach.

IEEE Journal on Selected Areas in Communications

, 38(7):1600–1613, 2020b. doi: 10.1109/JSAC.2020.2999696.

Giovanna Culot, Fabio Fattori, Matteo Podrecca, and Marco Sartor. Addressing industry 4.0 cybersecurity challenges.

IEEE Engineering Management Review

, 47(3):79–86, 2019. doi: 10.1109/EMR.2019.2927559.

Sahil Garg, Kuljeet Kaur, Georges Kaddoum, and Kim‐Kwang Raymond Choo. Toward secure and provable authentication for internet of things: Realizing industry 4.0.

IEEE Internet of Things Journal

, 7(5):4598–4606, 2020. doi: 10.1109/JIOT.2019.2942271.

Chengqiang Huang, Geyong Min, Yulei Wu, Yiming Ying, Ke Pei, and Zuochang Xiang. Time series anomaly detection for trustworthy services in cloud computing systems.

IEEE Transactions on Big Data

, 1, 2017. doi: 10.1109/TBDATA.2017.2711039.

Chengqiang Huang, Yulei Wu, Yuan Zuo, Ke Pei, and Geyong Min. Towards experienced anomaly detector through reinforcement learning.

Proceedings of the AAAI Conference on Artificial Intelligence

, 32(1), 2018. URL

https://ojs.aaai.org/index.php/AAAI/article/view/12130

.

Guozhi Lin, Jingguo Ge, Yulei Wu, Hui Li, Tong Li, Wei Mi, and E Yuepeng. Network automation for path selection: A new knowledge transfer approach. In

2021 IFIP Networking Conference

, 2021.

Xun Liu, Junling You, Yulei Wu, Tong Li, Liangxiong Li, Zheyuan Zhang, and Jingguo Ge. Attention‐based bidirectional GRU networks for efficient https traffic classification.

Information Sciences

, 541:297–315, 2020. ISSN 0020‐0255. doi: 10.1016/j.ins.2020.05.035. URL

https://www.sciencedirect.com/science/article/pii/S002002552030445X

.

Wang Miao, Geyong Min, Yulei Wu, Haozhe Wang, and Jia Hu. Performance modelling and analysis of software‐defined networking under bursty multimedia traffic.

ACM Transactions on Multimedia Computing, Communications, and Applications

, 12(5s):2016. ISSN 1551‐6857. doi: 10.1145/2983637. URL

https://doi.org/10.1145/2983637

.

Wang Miao, Geyong Min, Yulei Wu, Haojun Huang, Zhiwei Zhao, Haozhe Wang, and Chunbo Luo. Stochastic performance analysis of network function virtualization in future internet.

IEEE Journal on Selected Areas in Communications

, 37(3):613–626, 2019. doi: 10.1109/JSAC.2019.2894304.

Peijie Sun, E Yuepeng, Tong Li, Yulei Wu, Jingguo Ge, Junling You, and Bingzhen Wu. Context‐aware learning for anomaly detection with imbalanced log data. In

2020 IEEE 22nd International Conference on High Performance Computing and Communications; IEEE 18th International Conference on Smart City; IEEE 6th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)

, pages 449–456, 2020. doi: 10.1109/HPCC‐SmartCity‐DSS50907.2020.00055.

Christopher J. Turner, John Oyekan, Lampros Stergioulas, and David Griffin. Utilizing industry 4.0 on the construction site: Challenges and opportunities.

IEEE Transactions on Industrial Informatics

, 17(2):746–756, 2021. doi: 10.1109/TII.2020.3002197.

Guodong Wang, Yanxiao Zhao, Jun Huang, and Yulei Wu. An effective approach to controller placement in software defined wide area networks.

IEEE Transactions on Network and Service Management

, 15(1):344–355, 2018. doi: 10.1109/TNSM.2017.2785660.

Haozhe Wang, Yulei Wu, Geyong Min, Jie Xu, and Pengcheng Tang. Data‐driven dynamic resource scheduling for network slicing: A deep reinforcement learning approach.

Information Sciences

, 498:106–116, 2019. ISSN 0020‐0255. doi: 10.1016/j.ins.2019.05.012.

https://www.sciencedirect.com/science/article/pii/S0020025519303986

.

Haozhe Wang, Yulei Wu, Geyong Min, and Wang Miao. A graph neural network‐based digital twin for network slicing management.

IEEE Transactions on Industrial Informatics

, 1, 2020. doi: 10.1109/TII.2020.3047843.

Yulei Wu. Cloud‐edge orchestration for the internet‐of‐things: Architecture and AI‐powered data processing.

IEEE Internet of Things Journal

, 1, 2020. doi: 10.1109/JIOT.2020.3014845.

Yulei Wu. Robust learning‐enabled intelligence for the internet of things: A survey from the perspectives of noisy data and adversarial examples.

IEEE Internet of Things Journal

, 8(12):9568–9579, 2021. doi: 10.1109/JIOT.2020.3018691.

Yulei Wu, Hong‐Ning Dai, and Hao Wang. Convergence of blockchain and edge computing for secure and scalable IIoT critical infrastructures in industry 4.0.

IEEE Internet of Things Journal

, 8(4):2300–2317, 2021a. doi: 10.1109/JIOT.2020.3025916.

Yulei Wu, Hong‐Ning Dai, Hao Wang, and Kim‐Kwang Raymond Choo. Blockchain‐based privacy preservation for 5G‐enabled drone communications.

IEEE Network

, 35(1):50–56, 2021b. doi: 10.1109/MNET.011.2000166.

Yulei Wu, Zehua Wang, Yuxiang Ma, and Victor C.M. Leung. Deep reinforcement learning for blockchain in industrial IoT: A survey.

Computer Networks

, 191:108004, 2021c. ISSN 1389‐1286. doi: 10.1016/j.comnet.2021.108004. URL

https://www.sciencedirect.com/science/article/pii/S1389128621001213

.

Y. Wu, S. Singh, T. Taleb, A. Roy, H.S. Dhillon, M.R. Kanagarathinam, and A. De.

6G Mobile Wireless Networks

. Springer, 2021d).

Zhongxia Yan, Jingguo Ge, Yulei Wu, Liangxiong Li, and Tong Li. Automatic virtual network embedding: A deep reinforcement learning approach with graph convolutional networks.

IEEE Journal on Selected Areas in Communications

, 38(6):1040–1057, 2020. doi: 10.1109/JSAC.2020.2986662.

Shu Yang, Laizhong Cui, Xinhao Deng, Qi Li, Yulei Wu, Mingwei Xu, Dan Wang, and Jianping Wu. FISE: A forwarding table structure for enterprise networks.

IEEE Transactions on Network and Service Management

, 17(2):1181–1196, 2020. doi: 10.1109/TNSM.2019.2951426.

Juan Zhang, Yulei Wu, Geyong Min, Fei Hao, and Laizhong Cui. Balancing energy consumption and reputation gain of UAV scheduling in edge computing.

IEEE Transactions on Cognitive Communications and Networking

, 6(4):1204–1217, 2020. doi: 10.1109/TCCN.2020.3004592.

Yuan Zuo, Yulei Wu, Geyong Min, Chengqiang Huang, and Ke Pei. An intelligent anomaly detection scheme for micro‐services architectures with temporal and spatial data analysis.

IEEE Transactions on Cognitive Communications and Networking

, 6(2):548–561, 2020. doi: 10.1109/TCCN.2020.2966615.

2When Network and Security Management Meets AI and Machine Learning

2.1 Introduction

With the fast development of cloud computing, Internet of Things (IoT), and new‐generation mobile communication technologies (Wu, 2021, Wu et al., 2022, Cheng et al., 2020b), network services and systems are growing exponentially. Such an advancement of network technologies has brought about social progress, business innovation, and convenience of life, and at the same time, it inevitably causes the increase in the complexity of network management and security management. In this context, network and security management that is carried out manually or that which relies on automation rules are overwhelming, and there is an urgent need for help from emerging AI technologies (Wu et al., 2021). In this chapter, we will discuss why, how, and where AI technologies can empower network and security management.

Adopting artificial intelligence (AI) and machine learning (ML) is to realize automated and intelligent network and security management. The architecture usually considers a closed‐loop feedback iterative model of perception‐analysis‐control. The top‐down management channel is intent description, strategy selection, and configuration generation. The bottom‐up management channel is data fusion, behavior analysis, strategy selection, and configuration generation. The intelligence engine based on AI and ML is at the center of and is the core of the architecture.

In addition to the discussion around the architectural aspects, in this chapter, we also introduce several main types of ML techniques, including supervised learning, semisupervised/unsupervised learning, and RL, and provide examples of network and security management applications based on various ML techniques.

Further, we investigate and summarize the industrial applications and standards in the field of network and security management. The literature shows a significant growth of ML‐empowered products, demonstrating the significance and potential impact of AI and ML in the field of networking. However, how to leverage ML technologies efficiently, accurately, and stably in realistic networks with uncertainties is still an area worthy of research.

The remainder of this chapter is organized as follows: Section 2.2 introduces the architecture of ML‐empowered network and security management. Sections 2.3–2.5 introduce existing network and security management applications based on different types of ML techniques. Sections 2.6–2.9 summarize the industry products, standards, projects, and proof‐of‐concepts on network and security management. Finally, Section 2.10 concludes this chapter.

2.2 Architecture of Machine Learning‐Empowered Network and Security Management

Nowadays, network and security management relies heavily on human experience and skills. Telecommunication network is becoming increasingly complex under the heavy historical burden. Especially with the advent of 5G and cloud/edge computing, the scale of the network has increased by 10 times, and the network traffic is showing great uncertainties, which has exceeded the reach of human professional knowledge and ability. Facing the future, a large number of real‐time services are beyond the reach of human response speed. At the same time, the quality of network services is constantly improving. For example, services can be accessed quickly, network faults can be eliminated in a timely manner, and end‐to‐end (E2E) service quality can be guaranteed. Therefore, industry puts forward clear requirements for the intelligentization of the network. The current device‐centric network operation and maintenance model has been difficult to effectively achieve the above aims. Only by building a truly user experience‐centric network can it effectively meet the demands of end users and support the commercial success of network service providers. With the maturity of cloud computing, big data, and AI technologies, network solutions driven by user intents have emerged. It builds an intelligent network brain between the physical network and the business intent to achieve user experience‐centric networks.

This section presents the overall architecture of the application of AI and ML in network and security management. The technical idea is to adopt a closed‐loop feedback iterative model of perception‐analysis‐control. As shown in Figure 2.1, the architecture consists of a smart brain layer empowered by AI and ML techniques and a network infrastructure layer.

Figure 2.1 The overall architecture of AI‐ and ML‐empowered network and security management.

The smart brain layer aims to achieve the intelligent management and control of the network. It accepts the business intentions of operators and accurately transforms the intentions to network strategies. The smart brain automatically verifies and implements the strategies to physical networks. It perceives the states of the physical network in real time, detects anomalies, and issues early warnings in time to provide advice on how to handle anomalies and ensure that the network meets the business intents. It can quickly troubleshoot anomalies or optimize network performance based on the telemetry data and the experience knowledge graph database. Through continuous modeling and network behavioral learning, it can identify, e.g. short‐term congestion points that are prone to occur in advance and migrate important services or improve scheduling strategies.

The smart brain layer has five functional modules, including an intent engine, a strategy automation deployment engine, a data and behavior analysis engine, a knowledge graph engine, and a security‐monitoring engine. The intent engine is responsible for receiving and translating business intents into network strategies, simulating and verifying network strategies. The strategy automation deployment engine transforms network strategies into specific network commands and delivers it to network devices automatically through standard interfaces. The data and behavior analysis engine collects and analyzes network data and behaviors based on real‐time telemetry technologies. It verifies whether the current network conforms to the user's intents through AI techniques. These analysis results are also used to construct a knowledge graph. The knowledge graph engine continuously improves the experience graph database of network knowledge. Taking advantage of the knowledge graph, it helps the intent engine to continuously optimize the ability of intents translation. Based on real‐time network data and knowledge graphs, risky events are predicted and the corresponding mitigation suggestions are given. The security monitoring engine is responsible for performing security threat analysis based on network behavior data, identifying abnormal network traffic, and monitoring the entire life cycle of the network.

The smart brain layer is the central nervous system of the network. It achieves the data acquisition, verification, translation, and distribution of user intentions in a top‐down manner, so as to build a user‐centric automation and intelligent network. Its bottom‐up channel is to use AI techniques to analyze various measurement data and service behaviors of the entire life cycle of networks to achieve the automated and intelligent network and security management.

The network infrastructure layer is the cornerstone. It needs to meet the high bandwidth demand brought by new services such as 4K/virtual reality (VR) and other ultra‐high‐definition video, 5G, as well as autonomous driving. The network architecture continues to be restructured and optimized. The network can provide elastic scalability, plug‐and‐play capabilities, and provide deterministic low latency for future 5G services and industrial special applications, which is to serve different business needs.

2.3 Supervised Learning

Supervised learning is the process of using a set of samples with known labels to learn appropriate parameters and construct a learner. Therefore, the learner can map the newly arrived unlabeled data to the category (label) to which it belongs. The core of supervised learning is to label data, that is, learning from historical experience. Typical examples of supervised ML applications are shown in Table 2.1.

2.3.1 Classification

Classification is a typical supervised learning process. Based on the known categories, what the classification needs to do is to classify each unlabeled data into the corresponding category. The information of each category must be known in advance, and all data items to be classified have the corresponding categories by default. Thus, classification algorithms have limitations on whether they can meet the above conditions. Common classification algorithms include decision trees, support vector machine (SVM), K‐nearest neighbor (KNN) and neural networks, to name a few.

Table 2.1 Examples of supervised ML applications.

Applications

Works

Features

Traffic analysis

Peng et al. (

2016

) and Soysal and Schmidt (

2010

)

Compare multiple machine learning classifiers applied to traffic classification

Goseva‐Popstojanova et al. (

2014

)

Expose common threats in the Web to be identified

Belavagi and Muniyal (

2016

)

Compare several supervised machine learning classifiers for intrusion detection

Anomaly detection

Liu et al. (

2015

)

Automatically select the appropriate detector‐parameter combinations and the thresholds

Log analysis

Liang et al. (

2007

)

Train three classifiers for log prediction

Zhang et al. (

2019

)

Automatically learn the importance of different log events for anomaly detection

Li et al. (

2018a

)

Suggest the most appropriate level for newly‐added logging statements

Farshchi et al. (

2015

)

Find the correlation between the logs and the effect of operation activities on cloud resources

He et al. (

2018

)

Detect the cause of system performance degradation

The idea of decision trees is to use a tree structure to classify. Taking binary classification as an example, it is expected the model that learns from a given labeled dataset can classify new data. This task of classifying samples can be regarded as a “decision” for the question of “is the current sample normal?” Essentially, the decision‐making process of decision trees is the process of “decision” for a certain attribute of a sample. Due to the visualized tree structure, the decision tree has strong interpretability and has low requirements on the scale of training data. However, decision tree algorithms may generate very complex tree models, and the decision tree is usually based on a heuristic algorithm, so it may be unstable. Commonly used decision tree algorithms include Iterative Dichotomiser three (ID3) (Quinlan, 1986), C4.5 (Quinlan, 1993), Classification and Regression Tree(CART) (Breiman et al., 1984), etc.

One of the ideas of classification is to find a partitioning hyperplane in the sample space based on the labeled training dataset to separate samples of different categories. The idea of SVM is to find the maximum margin hyperplane with the best tolerance. The classification idea of SVM is simple, and the performance is usually good. SVM using the kernel function, can solve nonlinear classification issue, and it is suitable for classification tasks with less‐training data. However, when the data scale is large, the space and time overhead of SVM increases, and SVM is mainly suitable for binary classification problems.

KNN has a simple idea, that is, each sample can be represented by the KNN samples. If most of the KNN samples belong to a certain category, the sample also belongs to this category. KNN uses the Euclidean distance as a distance metric measuring two samples. KNN is simple and easy to use and implement, but it is easily affected by the configuration of and the impression of noise.

Generally, neural networks refer to the artificial neural network, which is an algorithmic mathematical model that imitates the behavioral characteristics of animal neural networks and performs distributed parallel information processing. This kind of networks relies on the complexity of the system and achieves the purpose of processing information by adjusting the interconnection between a large number of internal nodes. Neural network algorithms are based on a set of interconnected input and output units, and each connection between the units is associated with a weight. In the learning phase, neural network algorithms adjust the weight to achieve the correspondence between the input sample and its category. The neural network classification algorithm is accurate and flexible and can be adapted to various types of data, but correspondingly, the time and space overhead of neural networks is much greater than other types of classification algorithms.

Classification has been widely used in various scenarios and tasks of network and security management, such as anomaly detection (Liu et al., 2015, Wu et al., 2022, Guo et al., 2021, Huang et al., 2022, 2018), traffic classification (Rezaei and Liu, 2019b, Soysal and Schmidt, 2010, Cheng et al., 2020a–2021, Liu et al., 2020), log classification (Liang et al., 2007, Zhang et al., 2019, Sun et al., 2020, Zuo et al., 2020), etc.

The works in Peng et al. (2016) and Soysal and Schmidt (2010) compared multiple machine learning classifiers applied to traffic classification. For anomalous or malicious traffic detection, Goseva‐Popstojanova et al. (2014) exposed common threats in the Web to be identified. Belavagi and Muniyal (2016