Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

(1)

SDNを用いた大量電子メール攻撃の対策に関する研究

モハマド, ザフランビンアブドルアジズ

https://doi.org/10.15017/1931935

出版情報：Kyushu University, 2017, 博士（学術）, 課程博士バージョン：

権利関係：

(2)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

KYUSHU UNIVERSITY

GRADUATE SCHOOL OF INFORMATION SCIENCE AND ELECTRICAL ENGINEERING

DEPARTMENT OF ADVANCED INFORMATION TECHNOLOGY

DOCTORAL THESIS

Study on Mitigation of SMTP Flood Attack using Software Defined Network

MOHD ZAFRAN BIN ABDUL AZIZ

2018

(3)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Declaration of Authorship

I hereby declare that this thesis entitled " Study on Mitigation of SMTP Flood Attack using Software Defined Network” is the result of my own research except as cited in the references. This dissertation has not been accepted for any degree and is not concurrently submitted in candidature of any other degree.

Signature :

Student : MOHD ZAFRAN BIN ABDUL AZIZ Date : March 2018

Supervisor : Professor Koji OKAMURA

(4)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

To my family; parents, wife (Nazhatulsyima) and childrens (Aiman, Afeef,Ainul,Azhad

and Akira). I would not have done this without you. Thank you for your endless love,

encouragement, and support.

(5)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Abstract

This section briefly revisits the content of the thesis, which including the motivation for this work, novelty with respect to the previous work, problem definition, methodology, results and conclusions

Over the past decade, the Internet connectivity and its infrastructure are being attacked

by many sophisticated techniques such as intelligent denial of service browse attack, brute force,

shellshock attack, SSL attack, backdoor attack, botnet attack, SMTP flood attack etc. In this

work, the SMTP flood attack and spam mail was selected by the thesis for further research

investigation. Currently the SMTP Flood attack has been increased because many user in the

digital world are using free mail server. This inexperienced user accessing their free email

account thru portable devices such smartphone and tablet whereby most of these devices are

not equipped with antivirus, firewall or internet security suites (e.g. AVG, Norton, Avira). This

absences has exposed the communication devices with various vulnerability over the unsecure

network or the Internet. The SMTP flood attack is a form of net abuse, which an attacker send

huge volumes of email to one email address by intention to overflow the mailbox or overwhelm

the receiving mail server (e.g. @yahoo, @gmail) with a denial-of-service (DoS) attack. To

overcome this threat, the network security devices and software such as Network Intrusion

Detection System (NIDS) and Intrusion Prevention System (IPS) are continuously updated to

identify and prevent new threat. These systems are developed by many business entities and the

devices are usually protected by international law such as copyrighted and patented operating

system and software. By using different types or brands of network equipment, this has forced

an organization to employ a specialist on variety of network equipment. Diversity of

(6)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

configuration on different systems have also increased the possibility of network misconfiguration which is highly dangerous circumstances for the organization. Thus, there is a need for new paradigm and technology to make networks more scalable, dynamic and to allow easier management of network devices from different vendors. Thus, there is an urgency for new innovation to make network infrastructures more adaptable, ease configurable, scalable monitoring and less administration demanding from various devices by diverse vendors. These necessities could be fulfilled by programmable networks which can be realised using Software Defined Networking (SDN).

Currently, SDN is ongoing developing technology that tracts consideration because of

its worldview (abstraction system). It allows the control plane (top abstraction layer) to control,

monitoring and sharing information of various network devices (e.g. router, NIDS) using high

programmability enable function. Many preceding research work has believed that it could

replace traditional networking. By having SDN, the differences in proprietary operating

systems or interfaces of network devices is eliminated and makes the network administration

works less burden and independent from data plane vendors. SDN also enables rapid

development of network applications from developers that can be part of the single control

plane which can be managed from a centralized interface and have access to all connected

devices. The control plane act as the network operating system which has the ability to control

the states of all connected network. Data plane, in the other hand, is to receive the instructions

from the control plane and simply just forward the data packets. Currently, a standard data plane

abstraction such as Openflow, enable the use of any type of data plane devices that available

since all of the connected network devices can be manage by a common open source protocol.

(7)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

OpenFlow protocol is a clean slate project introduced by Stanford University back in year 2008. OpenFlow was implemented as the first open standard interface SDN architecture.

In Openflow, the data path and the high level routing decision are made from two different devices, which is the OpenFlow enabled switch and controller, respectively. The central controller provides the switches with the operational rules instructions, which is pushed by the controller to the switch as individual or group flow entries via a secured channel between them using OpenFlow protocol. Furthermore, Openflow actions function can be performed on network flows that can be forward and drop packets or manipulating the packet header information. The adjustability provided by SDN and Openflow create a sudden spark of new research area and applications such as application aware network, network flow virtualization, real time QoS support, helping Internet of Things (IoT) to reduce the strains of data generated by IoT devices, new ways to detect DDoS attacks any many mores. Researchers believed that a large number of new network applications will be introduced which will enhance the current network operations.

Open Network Operating System (ONOS) is the SDN network operating system for

service providers architected for performance, high availability, scale-out and well-defined

northbound and southbound abstractions and interfaces. ONOS was open-sourced on Dec. 5,

2014. In the spirit of freedom and openness, ONOS logo is a bird and the ONOS releases are

named after birds in alphabetical order. The first open source release of ONOS was called

Avocet and the next is Blackbird. Blackbird, which was released recently focuses on

performance optimizations, defining metrics for measuring the “carrier-grade quotient” of SDN

control planes/controllers and publicly providing the measurements for ONOS using these

metrics. The ONOS ecosystem comprises ON. Lab, organizations who are funding and

contributing to the ONOS initiative including Tier 1 service providers -AT&T, NTT

(8)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Communications, SK Telecom, leading vendors Ciena, Cisco, Ericsson, Fujitsu, Huawei, Intel, NEC; members who are collaborating and contributing to ONOS include ONF, Infoblox, SRI, Internet2, Happiest Minds, CNIT, Black Duck, Create-Net, KISTI, KREONET, KAIST and the broader ONOS community.

Firstly, this thesis has proposed an analysis of botnet attack for SMTP server using Software Define Network .This work provide the information about SMTP Flood attack traffic using SDN as new platform to do mitigation over the SMTP Flood Attack. Most of the information being analyst such as roundtrip time, 3 way handshake and Time to live data. This work will help to enhance the study on botnet attack behaviour.

Secondly, this thesis proposed a design of FlowIDS framework that is able to enhance

the existing network intrusion detection system (suricata) for more accurate detection and

mitigation in SDN to improve IDS framework for network security services. The first stage is

to check the SMTP traffic flows against the existing flow based signature for known SMTP

traffic flow attacks. If known attacks are mounted, it will update SDN (e.g. ONOS) to drop the

SMTP traffic flows. For the stage two, a flow-based detection is used to detect unknown

anomaly for SMTP traffic flows. To improve for a real-time detection, FlowIDS will distribute

the stage two work into multiple distributed computing systems. This will reduce computing

processing and loading if the FlowIDS is run on the same machine (or virtual machine) with

the NIDS. It also provides load balancing for processing huge SMTP traffic flows. If the stage

2 has detected an attack, it will update SDN to drop the SMTP traffic flows and also update the

flow based signature (stage 1) for a future signature attack detection. If the SMTP traffic flows

passed the stage 2, it will update SDN for legitimate SMTP traffic flows.

(9)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Thirdly,this thesis proposed a method to identify legitimate SMTP flow using Deep Learning and Decision Tree to increase network bandwidth free during SMTP flood attacks on single-site. In this work, we have used classification decision tree (DT) and deep learning (DL) algorithm to identify legitimate SMTP traffic flow which can be used to detect the SMTP flood attacks on the same malicious dataset. The proposed method will employ the legitimate SMTP flow value in our algorithm in our FlowIDS framework. The framework to detect SMTP attack using FlowIDS in a single site SDN. The outcome of the simulation is to enhance anomaly detections of SMTP flood attacks over single-site SDN. The implementation using Decision Tree Classification and Deep Learning algorithm is defined as “a predictive modelling technique from the fields of machine learning and statistics that builds a simple tree-like structure to model the underlying pattern, as Deep Learning approach that consists of stacked sparse auto encoders and softmax classifier for unsupervised feature learning and classification.

After the dataset being train by machine learning, the resulting model must be analysed. The accuracy of the model and the insights gained from resulting model are important. Model accuracy is usually straightforward to measure using techniques such as k-fold cross validation can test the model’s accuracy in a meaningful way.

Lastly, this thesis proposed a method to collaborate and mitigate SMTP flood attack on

multi-site using SDN. These proposals take advantage of Open Network Operating System

(ONOS) platform to advance the Multi-Domain ONOS Provider (MDOP) enables several

ONOS clusters to share information about their networks focus on SMTP traffic flow. For early

detection about attack, method using multi-site mitigation are well suggested, in research

industry several name have been introduce such as a collaborative protection network. The

major important in this work is the information that sharing with the other collaborative, in this

thesis, the simulation using multi-site network environment topology. If the collaborative

(10)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

become bigger and many node or switch. More accuracy of the anomaly flow can be detect at

near to source attack will result earlier recovering time can be execute.

(11)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Acknowledgements

All praise is due to Allah almighty God for all the graces, and blessings bestowed upon my family and me during my study in Kyushu University and living in Fukuoka, Japan. The process of earning a doctorate and writing a dissertation is long and arduous. It is certainly not done singlehandedly and beyond question, without guidance, loves and sacrifices.

First and foremost, I would like to thank my wife (Nazhatulsyima) and family (Aiman, Afeef, Ainul, Azhad and Akira) for putting up with an absentee husband and father during this process. Nazhatulsyima has been unfailingly supportive – and has borne the burdens which have fallen on her shoulder as I spent my most of the time and energy pursuing goals that took me away from her and the family.

I would certainly be remiss to not mention and sincerely thank my Professor, supervisor and mentor Professor Koji OKAMURA, whom without his help, wisdom, guidance, expertise and encouragement, this research would not have happened.

I also would like to show my deep gratitude to my loving parents, who have been my first teachers as they taught me about life, and have always been encouraging me to achieve more and do my best.

In addition, I would like to express my gratitude to Professor Yasuo OKABE, Professor Akihiro NAKAO, Professor Sachio HIROKAWA and Professor Hiroshi KOIDE for their kind support, guidance, and enlightening discussions throughout the meetings along my PhD study. Meetings they attended were really pleasant and fruitful, thanks to their kind comments, and advice that helped a lot to shape my work.

And also I would like to express my gratitude to Professor Sanjay Jha , Dr Guillaume Jourjon and laboratory members in CSIRO Sydney Australia for theirs kind accompany, comments, encouragement, and support.

I would like to express my gratitude towards all members of Professor Okamura’s

laboratory; Dr Masri, Dr Chengming Li, Dr Alaa Allakany, Mr Ariel, Mr Tam and all others

(12)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

that have their place in my memory. Thanks a lot for your kind accompany, comments, encouragement, and support.

Last but not least, I would like to thank the Ministry of Higher Education of Malaysia

and University of Technology MARA Malaysia for supporting my study in Japan. UNSW for

support my field research in CSIRO Sydney Australia and I would like to thank Kyushu

University, and the Graduate School of Information Science and Electrical Engineering, and all

their Professors and Officials for their kind support.

(13)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

List of Figures

Figure 1.1 SDN Architecture ... 21

Figure 1.2 SDN A high level architecture [8] ... 28

Figure 1.3 SDN Stacks [9] ... 28

Figure 1.4 Structure of deep learning (Deep Belief Network) ... 38

Figure 1.5 Push Back Method illustration ... 40

Figure 3.1 Integrated Spamhaus in multi-domain SDN ... 47

Figure 3.2 Decision Tree ... 48

Figure 3.3 Botnet Attacks Mitigation Process Algorithm ... 49

Figure 3.4 A flow graph of SYN flood ... 50

Figure 3.5 A flow graph botnet for SYN flood (comment) ... 51

Figure 3.6 A total of SMTP packets per second on 13 Jun 2010 ... 51

Figure 3.7 A total of SMTP packets per second on 15 Jun 2010 ... 52

Figure 3.8 A summary of max RTT and RTO for seven traffic datasheets ... 52

Figure 3.9 A graph of average TTL for packet for seven traffic datasheets ... 54

Figure 4.1 ONOS architecture [6] ... 58

Figure 4.2 A decision tree of content blind technique [16] ... 59

Figure 4.3 CTDA architecture for anomaly detection ... 60

Figure 4.4 Dossy framework for mitigating DoS attacks ... 63

Figure 4.5 An overview of FlowIDS framework ... 64

Figure 4.6 A process to detect SMTP attack using FlowIDS ... 65

Figure 4.7 FlowIDS Algorithm ... 66

Figure 4.8 Experiment setup for SMTP attacks ... 70

Figure 4.9 SMTP Flood attacks are dropped at Switch 1 and 2 using FlowIDS in SDN ... 71

Figure 4.10 A summary of FlowIDS experiment (subcases 4) ... 72

(18)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Figure 4.11 A snapshot of parameter on dataset Internet traffic, ISCX University New

Brunswick (UNB) Canada [24] ... 72

Figure 4.12 The decision tree of legitimate flow ... 73

Figure 4.13 The accuracy of flow-legit for normal traffic ... 73

Figure 4.14 Table from conversion tool wireshark ... 74

Figure 4.15 Experiment results of the four subcases ... 76

Figure 4.16 Dossy packet blocking (drop) behaviour [57] ... 76

Figure 4.17 FlowIDS using Deep Learning (DT) packet blocking (drop) behaviour ... 77

Figure 5.1 FlowIDS framework [17]... 82

Figure 5.2 FlowIDS process flow ... 83

Figure 5.3 An example of DT for SMTP attack detection (decision rules) ... 84

Figure 5.4 Classification of deep learning methods ... 85

Figure 5.5 Deep Learning (Deep Belief Network) ... 87

Figure 5.6 FlowIDS(DL) detailed process ... 88

Figure 5.7 Experiment setup for SMTP single site SDN Mitigation SMTP Flood attack ... 93

Figure 5.8 SMTP Flood attacks are dropped at Sites 1 and 2 using FlowIDS in a single site SDN ... 93

Figure 5.9 Tensor Flow Cloud based tool ... 94

Figure 5.10 Prediction table Deep Learning training data result ... 95

Figure 5.11 Table from conversations tool in Wireshark... 95

Figure 5.12 A summary of FlowIDS experiment on single site SDNs ... 97

Figure 5.13 Result Comparison of FlowIDS with DT and DL algorithms ... 98

Figure 5.14 FlowIDS performance using DT and DL algorithms in a single site SDN simulation ... 99

Figure 6.1 Implemented Distributed NIDS in SDN Model ... 102

Figure 6.2 . A FlowIDS framework [14] ... 105

Figure 6.3 FlowIDS Algorithm ... 106

Figure 6.4 A process to detect SMTP attack using FlowIDS[14] ... 107

(19)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Figure 6.5 Experiment setup for SMTP multi-site attack. ... 109 Figure 6.6 SMTP Flood attacks are dropped at Sites 3 until 6 using FlowIDS in multi-site SDN ... 109 Figure 6.7 A summary of FlowIDS experiment (subcases 4) ... 111 Figure 6.8 A snapshot of parameter on dataset Internet traffic, ISCX University New Brunswick

(UNB) Canada [15] ... 112 Figure 6.9 The decision tree of legitimate flow for duration and flow rate value for multi-site ... 113 Figure 6.10 The accuracy of flow-legit for normal traffic for Multi-site ... 113 Figure 6.11 Experiment results of the four subcases multi-site SDN Mitigation SMTP Flood

attacks ... 114

Figure 6.12 Comparison between Single site and Multisite Mitigation Using FlowIDS ... 116

Figure 6.13 Sharing FlowIDS signatures between multi-site controllers ... 116

(20)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

List of Tables

Table 1.1 A summary of literature discusses D/DOS and spam attacks ... 32

Table 4.1 The summary of SMTP attack detection methods by IDS or firewall ... 61

Table 4.2 Simulation Parameter Table single site SDN ... 71

Table 5.1 Selected network flow attributes ... 89

Table 5.2 The accuracy of flow-legi for normal traffic (DL) ... 94

Table 5.3 Comparison accuracy score between DBN & DT ... 99

Table 6.1 Simulation parameter table for Multi-site SDN ... 110

(21)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Chapter 1

1. Introduction

Nowadays, with the constant development of informational technology, email become important in term of verification process. In the role of cyber security gets ever bigger. While it is impossible to imagine modern world without constant communications over the network, almost all valuable data that often acts as a target for attacks, is stored in various forms on the servers. Not to mention that the stability of the whole system depends on the servers. This is why the servers are very attractive targets for malicious attacks.

Security of mail servers, among others, is a particularly important question. This question is important because email is one of the most popular means of communication and doing business. And for businesses in particular, loss of confidential information can result in large financial loses. It is also important for the server to run stable, so that users are able to access it at any time. When the server runs unstable, it can lead to the loss of customers.

The volume and sophistication of attack traffic has increased dramatically over the years [1]. As a result, the ability to protect a network from threats is of paramount importance to network operators. The current popular techniques for securing the network include firewalls, Access Control Lists (ACL), Anti-Virus gateways or client-side agents (AV), encryption and Intrusion Detection and Prevention systems (IDS/IPS). Many of these are implemented in the network path at a central route point, so that the majority of network traffic can be examined and filtered. Most of these systems are relatively easy to deploy and maintain in an effective state when in small deployments.

For email spam mail attack only focus on sending unsolicited message , commercial

message that contain disguised links that appear to be for familiar websites but in fact lead to

phishing web sites or sites that are hosting malware. These problem however can be settle by

training the user about precaution against email spam. But for SMTP flood attacks, they are

targeted to bring down any email server that import to any organisation or company. In these

case a solution need to be investigate.

(22)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

1.1. Background

However, the IDS/IPS devices and software have struggled to reach an effective working status in many large organizations due to their complexity and pricing related capacity constraints. These devices are designed to provide alerting and auditing of malicious activity by monitoring network traffic in real-time. An IDS/IPS will be configured to examine network packets for known bad patterns and traffic pattern anomalies, in either detect mode in the case of an IDS and in a prevent mode in the case of an IPS.

Additionally, the required data throughput capacity is also of significant importance.

Even in the simplest configuration, these systems are well known for flooding network operators with false positives [2], thereby degrading their effectiveness and and as the requirements of users change these false positive grow in number. The primary constraint is throughput, as network operators will need to deploy an IDS/IPS that has sufficient capacity to avoid buffering in IDS mode or packet delay in IPS mode. As with most network devices, the greater the capacity the greater the cost, often to a level that strains IT budgets.

The use of Software Defined Networking (SDN) allows a level of service interaction and decentralization that has not previously been available in the traditional network model.

Where most networks rely on static pre-defined configurations, a software-defined network can be a dynamic, highly customizable alternative that is designed to adapt to changing requirements.

The interest in SDN has increased significantly in recent years, not only in research but

also in the commercial aspects [3]. As its popularity has increased so too has the interest in its

use across a wide range of use cases, from Quality of Service (QoS) and access control to load

balancing and service provisioning. Part of this success is due to the use of open standards such

as OpenFlow [4] and HTTP for the REST API, this has allowed the easy integration of third

party products and services. Combining the extensibility of SDN with the filtering and threat

management of IDS will allow operators to provide threat management at any point in the

(23)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

network. In particular, the ability to cost-effectively deploy filtering at the point of ingress would significantly reduce the impact of incidents.

The key question we seek to answer in this paper is whether the SDN architecture can provide an IDS function within the SDN framework, or whether the IDS function should be provided by a traditional architecture. The long goal of this research, which will require many other research questions to be answered, is to produce an SDN/IDS solution that can actively adapt to network usage patterns, thereby providing an agile, scalable, highly responsive, device targeted security management system with far fewer false positives. Ultimately the new architecture will aim to achieve the exclusive “working effectively” audit status.

Limitation of current network technologies related to our research have bring major problem such as inconsistent policies, inability to scale and vendor dependence. Our research related to policies such as to implement a network-wide policy, IT may have to configure thousands of devices and mechanisms. For example, every time a new virtual machine is brought up, it can take hours, in some cases days, for IT to reconfigure ACLs across the entire network. The complexity of today’s networks makes it very difficult for IT to apply a consistent set of access, security, QoS, and other policies to increasingly mobile users, which leaves the enterprise vulnerable to security breaches, noncompliance with regulations, and other negative consequences.

1.1.1. SMTP Flood Attacks

N. Hoque et al. (2014) [7] discuss tools used by attackers and security admin in SDN.

The authors revisit machine learning algorithm, flow-based features for botnet detection using a predefined dataset. The dataset consists of SMTP Flood attack flow and UDP Storm and it successfully detected with rate 75%. S. Lim et al. (2014) [8] propose to utilize SDN for DDoS attack detection and prevention. The authors discuss a method to block the DDoS attack using OpenFlow in SDN controller. It was simulated in POX controller using Mininet emulator. C.

Schafer [9] (2014) uses geolocation and country to detect an anomaly that can be used to

(24)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

identify spam email. A novel contribution, Theoretical Geographical Travelling Speed (TGTS) method is proposed in his work. T. Sochor (2014) [10] revisited the existing methods to detect and prevent spam messages. Multi-layer protection technique such as blacklisting and greylisting was discussed. E. B. Beigi et al. (2014) [11] re-examined flow-based for botnet detection, which also studies its effectiveness in detection using a predefined dataset. T. Ouyang [12] et al. (2014) study spam filtering pipeline for finding its accuracy and trade-off in four layers. The authors used three decision trees: packet features, flow features and the combination of both features. Figure 3 show example of decision tree for spam email detections.

H. Chen et al. (2015) [13] integrate entropy measurement for flooding detections in mail systems. It studies an entropy in round-trip time (RTT) and retransmission timeout (RTO) to detect dangerous traffics. The entropy can help to improve malicious mail analysis and detection for protocols: SMTP, IMAP4, POP3 and HTTPS

1.1.2. Software Define Network

SDN is an architecture for multi devices communication in integrated networks. In the initial stage, it allows multiple LANs devices and systems to be integrated into WAN networks.

The first SDN began after Java language released by Sun Microsystem, which AT&T Labs Geoplex project used Java to program APIs to implement middleware networking [1]. The Geoplex provided open networking standard for network integrations and communications such as system managements and provisions, integrated security and system authentication, network monitoring etc. By 2011, Open Networking Foundation (ONF) develops OpenFlow for SDN [2]. The ONF provides SND resources (e.g. switch specification) for product manufacturer and software developer to implement SDN using the OpenFlow’s standard and protocol [3].

Figures 1.1 show a general SDN architecture and its stacks. In SDN topology, all

network nodes or devices are controlled using a control plane. The architecture splits the control

plane from actual network data and routing process (data plane). The infrastructure layer

communicates with SDN Controller using Control Data Plane (CDP) API (e.g. OpenFlow). All

(25)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

nodes or routers in the SDN network will use the CDP API for all control plane communication.

The control layer consists of SDN Control Software or Controller, which extract information from the infrastructure layer such as a list of all devices in the SDN network and its states. It does not provide the entire information of all connected devices, but it provides an abstract view of the SDN network and topology. The application layer uses information from the control layer for a network abstraction administrative such as network analytics; network, system and topology managements etc.

Figure 1.1 SDN Architecture 1.1.3. Open Network Operating System (ONOS)

ONOS is the SDN network operating system for service providers architected for performance, high availability, scale-out and well-defined northbound and southbound abstractions and interfaces. ONOS was open-sourced on Dec. 5, 2014. In the spirit of freedom and openness, ONOS logo is a bird and the ONOS releases are named after birds in alphabetical order. The first open source release of ONOS was called Avocet and the next is Blackbird.

Blackbird, which was released recently focuses on performance optimizations, defining metrics

(26)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

for measuring the “carrier-grade quotient” of SDN control planes/controllers and publicly providing the measurements for ONOS using these metrics.

The ONOS ecosystem comprises ON.Lab, organizations who are funding and contributing to the ONOS initiative including Tier 1 service providers -AT&T, NTT Communications, SK Telecom, leading vendors – Ciena, Cisco, Ericsson, Fujitsu, Huawei, Intel, NEC; members who are collaborating and contributing to ONOS include ONF, Infoblox, SRI, Internet2, Happiest Minds, CNIT, Black Duck, Create-Net, KISTI, KREONET, KAIST and the broader ONOS community.

1.1.4. Network Security

Distributed systems such as cloud computing and Internet of Things (IoT) are not the main factors for organizations to migrate theirs network infrastructure into SDN, another main reason is a network security that offered by the SDN [15, 16]. The SDN allows an abstraction of network security that provides a central authority in a network, which previously hard to be done by traditional distributed networking systems and infrastructures [4, 5]. There are also new security problems introduces by an implementation the SDN in network infrastructure, but we are not going to discuss in this publication and one may refer to [16–19] for further examinations regarding these security problems. The following paragraphs will discuss security threats and its countermeasures using SDN.

N. Hoque et al. [20] discuss tools use by attackers and network administrators in SDN.

Major attacks on SDN are Dos and DDos [21] that mounted by botnets [22]. Most botnets will

try to prevent access to computing resources in the SDN by draining computing capability of

the target computing system. An attacker(s) frequently used SYN-Flooding Attack [23], which

sends a flood of TCP/SYN packets (by zombie machines) and leave the 3-ways TCP handshake

protocol hang-up without ACK packets. This attack applied to all application protocols that are

used TCP based connections such as SMTP, FTP, HTTP, DNS etc. Traditional network security

systems and infrastructures rely on Intrusion Detection System (IDS) and firewall to protect

(27)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

LAN, WAN from the internet. It might work well for a small and manageable network such as LAN, but not for multi-WANs in a large organization (or a join of multiple organizations) in distance geographical locations. Furthermore, applying SDN for the entire internet is far away than a current topic, which requires, at least a successful implementation of SDN for multi- WANs. We skipped this part, but we want to narrow down our discussion that to improve an efficiency for botnet attack detections on SMTP protocol. The next paragraph will explore the existing methods in preventing the botnet attacks on SMTP protocol.

The most common way to detect botnet attacks are using a signature-based of known attacks [24], and a real-time detection of network anomalies [24, 25] using IDS. Both methods used congestion control and drop packet to block DDos attacks, which called Pushback method [21]. The signature-based requires others systems to provide the signature of known attacks, which can be derived from the real-time detection from a shared database. Routers within the same LANs/WANs may share or distribute attack signatures, for examples a list of blacklisted source and destination IPs, payloads, Time-to-Live (TTL) [26] etc. Another method to detect potential attacks is using a network traffic classification. It can help to identify packets send by botnets at local and enterprise networks [27]. This method may be integrated into the real-time detection method.

1.2. Motivation and Goals

Software Define Networking (SDN) and Openflow seems to be the frontier in Internet technology that enable to improve and create application development that were untapped in current traditional network. By having central controlling function, the management and network programmability is more practical and easy. In more attractive for future solution in network, a standard body call as open networking foundation (ONF) has dedicated to promote and adopt SDN technology through open standard development. Despite SDN decouple the control plane and forwarding plane separately and have more advantage than the traditional IP networking, it is still incomplete and has some challenges unsolved in certain area of

implementation.

(28)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

The first goal of this thesis is to provide analysis on botnet attack, DDOS and SMTP Flood Attack on SMTP Server and design a new framework for SMTP flood detection and mitigation using FlowIDS in SDN. The flexibility of SDN that allow to integrate with any third- party hardware, any network intrusion detection system software application and new framework will make the configuration more accuracy and consistency comparing with traditional network.Referring the most recently work using push back [18] method, the mechanism in which routers upstream of the server under attack are asked to start dropping packets to the server under attack all packet will drop include legitimate SMTP packet flow.

The proposed framework allowed to detect the SMTP flood attack using SDN based without drop the legitimate SMTP flow during the attacks. Based on previous work result using dossy packet blocking system [57], all packet were dropped until 0 % during the attack. FlowIDS is a framework for anomaly detection on SMTP traffic flows. The novelty of the FlowIDS is the detection method, whereby this work has introduced flow based attack detection on the SMTP traffic flows. It can be integrated with the existing network security systems such as firewall, IDS, SDN controller and ONOS application.

The second goal of this thesis is to enhance the detection and mitigation on SMTP flood

attacks using FlowIDS framework. In our work, we have used classification decision tree (DT)

and deep learning (DL) algorithm to identify legitimate SMTP traffic flow which can be used

to detect the SMTP flood attacks on the same malicious dataset. The proposed method to

increase network bandwidth free during SMTP flood attacks. In the previous work [17,113],

the analysis of botnet traffic and SMTP flood attack were done using decision tree (DT)

classification on pre-processing malicious dataset. Many method on empirical analysis of

SMTP protocol through network characteristic in stand-alone enterprise [31].A wide range of

approaches including expert system, pattern matching, state transition analysis, neural network

and statistics. A broad classification of these detection techniques places them into either

anomaly based or signature based methods. Anomaly based detection seeks to identify a normal

system state and detects deviations from that state as signs of anomalous activity. Signature

(29)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

based detection identifies intrusions by comparing a current state against known patterns, rules or states to recognize an intrusion.

The final goal of this thesis is to utilize centralize system that provided in SDN allow the centralize information regarding network behavior according to its needs. In case of multi- site mitigation against anomaly attacks ,the sharing information regarding anomaly attacks between sites, domain and multiple DMZ zone, for example in a network enterprise it could be between many branches or in a university between many campuses. By having real time anomaly attacks information in many site, and this information being centralize and process, it may help the mitigation on any anomaly attacks more efficient decision making. In the previous work [111,112] the mitigation was done in single site or standalone network [18] against SMTP flood attack on SMTP server. Our work proposed a method that allows to collaborate and mitigate the SMTP flood attack on SMTP server close to the source of attacks in other site network topology (early mitigation) by update the rules to all SDN controllers within multi-site network.

The reminder of this thesis is organized as follows. Chapter 2 illustrates technical

background and network architecture on SDN, Network Security and SMTP Flood Attack. An

analysis of botnet attack for SMTP server using SDN based is presented in Chapter 3. In

Chapter 4 a method to detect SMTP flood attacks using FlowIDS framework. A comparison

between Decision Tree and Deep Learning Algorithms on Collaborative Mitigation SMTP

Flood Attacks in SDN Topology is proposed in Chapter 5. In Chapter 6 collaborative mitigation

SMTP flood attack using SDN platform on multisite. At last, Chapter 7 concludes all the works.

(30)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Chapter 2

2.0. Literature Review

This chapter provide more details briefly on the major topics touched in topics is the analysis of Botnet Attack for SMTP Server on Software Define Network Architecture . The analysis of botnet of flow sampling in anomaly detection techniques and the effectiveness of using SDN as a part of mitigation process against any attacks from botnet attack. To enhance the the mitigation process against Botnet Attack multi-site mitigation approach has been introduce, It is paramount important to understand the following parts of this study.

2.1. Architecture of SDN

SDN is an architecture for multi devices communication in integrated networks. In the initial stage, it allows multiple LANs devices and systems to be integrated into WAN networks.

The first SDN began after Java language released by Sun Microsystem, which AT&T Labs Geoplex project used Java to program APIs to implement middleware networking [1]. The Geoplex provided open networking standard for network integrations and communications such as system managements and provisions, integrated security and system authentication, network monitoring etc. The most prominent functionality of the Geoplex is it allows network IPs to be mapped to one or many system and services [2]. In 2008, research and development for SDN continue by UC Berkeley and Stanford University [3]. By 2011, Open Networking Foundation (ONF) continues to develop OpenFlow for SDN [4]. The ONF provides SND resources (e.g.

switch specification) for product manufacturer and software developer to implement SDN using the OpenFlow’s standard and protocol [5].

Figures 1.2 and 1.3 show a general SDN architecture and its stacks. In SDN topology,

all network nodes or devices are controlled using a control plane. The architecture splits the

control plane from actual network data and routing process (data plane). The infrastructure layer

communicates with SDN Controller using Control Data Plane (CDP) API (e.g. OpenFlow). All

(31)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

nodes or routers in the SDN network will use the CDP API for all control plane communication.

The control layer consists of SDN Control Software or Controller, which extract information

from the infrastructure layer such as a list of all devices in the SDN network and its states. It

does not provide the entire information of all connected devices, but it provides an abstract view

of the SDN network and topology. The application layer uses information from the control layer

for a network abstraction administrative such as network analytics; network, system and

topology managements etc. [6, 7].

(32)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Figure 1.2 SDN A high level architecture [8]

Figure 1.3 SDN Stacks [9]

(33)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

Many SDN runs over a virtualized architecture, which the application and control layers may execute in various devices that including a virtual machine in cloud computing [10, 11].

This allows application and control layers to be distributed on various computing platforms, which it will increase flexibility, mobility and computing power using the virtualized architecture, system and devices [12–14]. In this work, we will not discuss the advantage of SDN in distributed systems, but we want to assess a network security through SDN. The next subsection will discuss further the network security and threats in the SDN.

2.2. Network Security by SDN

Distributed systems such as cloud computing and Internet of Things (IoT) are not the main factors for organizations to migrate theirs network infrastructure into SDN, another main reason is a network security that offered by the SDN [8], [9]. The SDN allows an abstraction of network security that provides a central authority in a network, which previously hard to be done by traditional distributed networking systems and infrastructures [3], [2]. There are also new security problems introduces by an implementation the SDN in network infrastructure, but we are not going to discuss in this publication and one may refer to [9]–[12] for further examinations regarding these security problems. The following paragraphs will discuss security threats and its countermeasures.

M. Kim et al. (2004) [13] propose a prototype for abnormal network traffic detection. It uses an aggregation of packets that using identical flow, which can be to detect an attack when there are changes in traffic patterns. Changes in traffic patterns such as port number and payload can be detected using this method. B. Xiao et al. (2005) [14] present a method to identify attacker signature using an active probing scheme. It analyses delay in TTL for each router in networks, which are used to identifying SYN flooding detection. H. Luo et al.(2006) [15]

propose a method to detect SMTP anomaly using collection and aggregation of packet deviations. The method does not require to store all traffic data, it uses a leaky integrate-and- fire model (or weightage sum) to maintain previous traffic knowledge. S. Naksomboon et al.

(2010) [16] propose a behaviour method to filter known spammer’s behaviours in filtering rules.

(34)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

The authors used random forest algorithm and Spam Assassin Corpus database in their works.

G. Kakavelakis et al. (2011) [17] implement a botnet analysis using machine learning, SMTP Mail Transport Agent (MTA). It (or named SpamFlow) can be integrated into transport-layer for packet analysis. M. Still et al. (2011) [18] revisit the state of the art in DDoS for attack, detection, protection and mitigation in SMTP server.

R.K. Sahu et al. (2012) [19] perform performance analysis on DoS attack using SYN flooding attack. The results show that memory and CPU usage burst heavily when attacked. Z.

Duan [20] et al. (2012) monitor outgoing message by zombie machines for spam detection, which used sequential probability ratio test algorithm. P. Nevlud et al. (2013) [21] propose a method to detect network anomaly using machine learning systems. The authors used data mining framework, WEKA for network analysis, which used algorithms: decision tree and Bayesian networks. The purpose of this work is to find and build efficient knowledge structures of network attacks. V. Petkov et al. (2013) [22] investigate an entropy of aggregated network traffics for entropy fingerprints. M. Channegowda et al. (2013) [23] propose service on demand (SoD) architecture in SDN. S. Shin et al. (2013) [24] develop an OpenFlow security application (named FRESCO) for managing security modules in SDN. The FRESCO provides security architecture as well as API for security scripting and translation, database and event management, and instance execution.

N. Hoque et al. (2014) [25] discuss tools used by attackers and security admin in SDN.

The authors revisit machine learning algorithm, flow-based features for botnet detection using

a predefined dataset. The dataset consists of SMPT Spam and UDP Storm and it successfully

detected with rate 75%. J. Ioanidis et al. (2014) [26] show an implementation of Pushback router

to reduce DDoS attacks in a network. The implementation was done in FreeBSD. It uses

congestion control problems as a sign for DDoS attacks and flash crowds. S. Lim et al. (2014)

[27] propose to utilize SDN for DDoS attack detection and prevention. The authors discuss a

method to block the DDoS attack using OpenFlow in SDN controller. It was simulated in POX

controller using Mininet emulator. C. Schafer [28] (2014) uses geolocation and country to

(35)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

detect an anomaly that can be used to identify spam email. A novel contribution, Theoretical Geographical Travelling Speed (TGTS) method is proposed in his work. T. Sochor (2014) [29]

revisited the existing methods to detect and prevent spam messages. Multi-layer protection technique such as blacklisting and grey listing was discussed. E. B. Beigi et al. (2014) [30] re- examined flow-based for botnet detection, which also studies its effectiveness in detection using a predefined dataset. T. Ouyang [31] et al. (2014) study spam filtering pipeline for finding its accuracy and trade-off in four layers. The authors use three decision trees: packet features, flow features and the combination of both features.

H. Chen et al. (2015) [9] integrate entropy measurement for flooding detection in mail systems. It studies an entropy in round-trip time (RTT) and retransmission timeout (RTO) to detect dangerous traffics. The entropy can help to improve malicious mail analysis and detection for protocols: SMTP, IMAP4, POP3 and HTTPS. R. Sahay et al. (2015) [32] propose an implementation of a distributed collaboration framework for sharing information that can be used to mitigate DDoS in SDN. A client can request from Internet Service Provider (ISP) for a mitigation service by downloading (and installing) security middle boxes in client-side. The security middle boxes will perform attack detection and analysis at the client-side, while sharing collected information on attack with ISP. It can be done in autonomous communication, which to protect the entire SDN infrastructures from DDoS attacks. J. Jeong et al. (2015) [33] propose security services in SDN for a centralized firewall and DDoS mitigation systems. Y. Yan et al.

(2015) [34] review DDoS attacks on cloud computing and then how to prevent the DDoS

attacks by implementing SDN in the cloud computing. P. Holl (2015) [35] discusses multiple

methods to detect and prevent DDoS attacks in SDN such proactive and reactive defences, and

post-attack analysis. Q. Yan et al. (2016) [36] present a survey on SDN, DDoS in cloud

computing. The authors found that DDoS attacks on cloud computing are increasing because

of “On-Demand Self-Service Leading to Botnets Outbreak… Broad Network Access and Rapid

Elasticity Leading to More Immense, Flexible, and Sophisticated DDoS Attacks Resource

Pooling Leading to the Victims More Vulnerable to DDoS Attacks” [36]. To reduce such

(36)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

attacks, the authors suggest implementing SDN in cloud computing because it capable of dealing with dynamic network architectures.

The summary of literature reviews is presented in Table 1.1 shows many literatures that are not discussed in the previous paragraphs.

Table 1.1 A summary of literature discusses D/DOS and spam attacks

Authors Attacks Detection and Prevention

Kim [13] (2004) D/DoS, abnormal network traffic

Packets aggregation and identical flow

Xiao [14] (2005) SYN flooding Active probing scheme Luo [15] (2006) SMTP spam Weightage sum of traffics Bencsáth,[37] (2007) DoS, SMTP spam Content filtering applications

harm server performance Beverly [38] (2008) Spam Extraction of email TCP

features, SpamFlow A. Al-Bataineh [39]

(2009)

Reviews on: Botnet, spam

Monitoring outgoing traffic, DBSpam, SpamFlow, signatures

Hu [40] (2009) DoS and Worms Cisco NetFlow, entropy based adaptive flow

aggregation algorithm, detect source of attack in clusters Smith [41] (2009) Spambots Entropy of packets

distribution skewness Naksomboon [16]

(2010)

Email spam Spammer’s behaviors, random forest algorithm Ehrlich [42] (2010) Email spam, spam

bots

Network flow data, entropy- based traffic analysis Kakavelakis [17]

(2011)

D/Dos, SMTP spam Machine learning analysis Still [18] (2011) DDoS, zombie spoofs,

SMTP spam, SYN flooding etc.

Detections: pattern, anomaly and 3

^rd

party database.

Defenses: over provisioning, routing control, currency assumption and

authentication.

Push back.

Sahu [19] (2012) DoS, SYN flooding Exhaustive memory and CPU usage

Duan [20] (2012) DDoS, spam, zombie monitoring outgoing

(37)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

messages by sequential probability ratio test algorithm

Suwa [43] (2012) Botnet, spam mail DNS record characteristics and behavior of DNS servers, blacklist Nevlud [21] (2013) Abnormal network

traffic

Machine learning systems, WEKA, decision tree and Bayesian networks

Petkov [22] (2013) - Entropy fingerprints

Channegowda [23]

(2013)

- Service on demand (SoD) in

SDN

Lin [44] (2013) SMTP spam Tools: Bro IDS, Bloom filters

Jian-Qi [45] (2013) DoS Dynamic entropy-based

model for anomalies detection

Gada [46] (2013) Spam Dynamic whitelist in layer 3 switch.

Zempoaltecatl- Piedras [47] (2013)

Network anomaly Anomaly traffic filtering - Method of Entropy Spaces (MES), flow-level entropy space

Navaz [48] (2013) DDoS Combine entropy and

anomaly detection for multilevel DDoS, share signatures with Cloud Service Provider (CSP) Phemius [49] (2013) Network attacks DIstributed SDN Control

plane (DISCO) for WAN, analysis on inter-domain topology disruption

Shin [24] (2013) - Security framework in SDN,

FRESCO

Sochor [50] (2013) Spam Blacklisting

Phemius [51] (2013) Measure link latencies OpenFlow controller Cartier [52] (2013) DDoS, SMTP

flooding

Optimize connection timeout (TTL) configuration

Scott-Hayward [8]

(2013)

Survey on: SDN security

Attacks and security on SDN Rathi [53] (2013) Spam flooding Analyze existing methods

for spam mail detection e.g.

support vector machine,

naïve Bayes, decision tree,

feature selection,

(38)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

classification and prediction Hoque [19] (2014) Reviews on: SDN

security, botnet

machine learning algorithm, flow-based

Beigi [30] (2014) Botnet, DDoS, SMTP Spam, UDP Storm

Flow-based features Ioannidis [26] (2014) DDoS, flash crowds Congestion-control

problems, Pushback

Lim [27] (2014) DDoS POX/SDN controller,

Mininet

Schafer [28] (2014) Spam email Geo location and country Sochor [29] (2014) Reviews on: Spam

message

Blacklisting and greylisting Beigi [30] (2014) Botnet, SMTP spam Flow-based

Ouyang [31] (2014) Spam email Spam filtering pipeline.

Packet features, flow features and combined both Chen [54] (2014) Botnet, IRC Two-level correlation for the

same anomaly Vizváry [55] (2014) Reviews on: DDoS in

SDN

Malicious traffics Giotis [56] (2014) Network anomaly and

DDoS in SDN

OpenFlow with Remote Triggered Black-Hole (RTBH) routing Oktian [57] (2014) DoS, IP/MAC

spoofing, bulky/garbage message

Simulation of attacks by OpenFlow in Mininet, block all known unused resources in SDN

Hoque [25] (2014) Reviews on: attacks and defensive techniques in networks

Anomaly detection, scanning tools, attacking tools

Smeliansky [10]

(2014)

- Security by SDN

architecture.

Geetha, [58] (2014) DoS, SYN flooding attack

Analysis on SYN flooding attack.

Özçelik [59] (2015) Vulnerability in entropy based detection method

Attacker avoids detection

H. Chen [14] (2015) SYN flooding Entropy RTT and RTO Sahay [32] (2015) DDoS Distributed collaboration

framework in SDN, ISP and its client sharing information

Jeong [33] (2015) DDoS Centralized firewall and

mitigation systems in SDN

Graham [60] (2015) Botnet Flow export for

(39)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

identification botnet command control

Yan [34] (2015) DDoS Existing method to prevent

DDoS in SDN e.g. attacks on application, control and infrastructure layers Holl [35] (2015) DDoS, botnet Proactive and reactive

defenses, and post-attack analysis

Kim [61] (2015) DDoS Implementation of

centralized firewall and mitigation system in SDN

Seeber [62] DDoS Redirect identified suspicious

traffics to IDS for further inspection in SDN.

Yan [36] (2016) Survey on: DDoS in SDN and cloud computing

Implementation SDN helps to minimize DDoS in cloud computing – based on SDN features.

2.3. Traffic analysis on Botnet attack

Early works in botnet detection are predominantly based on payload analysis methods which inspect the contents of TCP and UDP packets for malicious signatures. Payload inspection typically demonstrates very high identification accuracy when compared with other approaches but suffer from several limitations that are increasingly reducing its usefulness.

Payload inspection techniques are typically resource intensive operations that require the

parsing of large amounts of packet data and are generally slow. Additionally, new bots

frequently utilize encryption and other methods to obfuscate their communication and defeat

packet inspection techniques. Furthermore, the violation of privacy is also a concern in the

payload analysis-based detection scheme. A more recent technique traffic analysis, seeks to

alleviate some of the problems with payload inspection. Traffic analysis exploits the idea that

bots within a botnet typically demonstrate uniformity of traffic behaviour, present unique

communications behaviour, and that these behaviours may be characterized and classified using

(40)

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

a set of attributes which distinguishes them from non-malicious traffic and techniques. Traffic analysis does not depend on the content of the packets and is therefore unaffected by encryption and there exists dedicated hardware which may extract this information with high performance without significantly impacting the network. Typical traffic analysis based detection systems examine network traffic between two hosts in its entirety. While this approach is feasible for offline detection, it is not useful for the detection of botnet behaviour in real time. A network flow between two hosts may run for a few seconds to several days, and it is desirable to discover botnet activity as soon as possible.

2.3.1. Proposed Research Objective

As discussed in previous sub-chapter (2.2 and 2.3), Network Intrusion Detection System (NIDS) has been used to detect the anomaly traffic such as, botnet attack, SMTP flood attack and DDOS. In order to overcome the problem and limitation listed above an approach to detect anomaly traffic in real time and mitigation against SMTP flood attacks using FlowIDS framework on SDN Based technology is proposed. FlowIDS is a framework for anomaly detection on SMTP traffic flows.

The novelty of the FlowIDS is the detection method, whereby this work has introduced flow based attack detection on the SMTP traffic flows. It can be integrated with the existing network security systems such as firewall, IDS, SDN controller and ONOS application.

2.4. Detection and Mitigation SMTP Flood Attack through Deep Learning analysis techniques in SDN

2.4.1. Traffic Identification

Traffic identification is a key component in network security since it raises the red flag in case, of intrusion into the network. Notably, the system has relied on traditional methods of detection that are increasingly becoming ineffective due to the commensurate increase in data.

Traditional approaches include port identification for instance, standard HTTP that is failing to

perform as envisaged due to less protocols following the system. Another system involves the