Summary

The thesis has presented a network security proposal to detect and mitigate SMTP flood attack over SMTP server in Software Defined Network Environment using our proposed FlowsIDS framework.FlowIDS framework can divide into two stages. The first stage is to check the SMTP flood attack against the existing flow based signature for known SMTP flood attacks. If known attacks are mounted, it will update SDN (e.g. ONOS) to drop the SMTP traffic flood attack flow. For the stage two, a flow-based detection is used to detect unknown anomaly for SMTP traffic flood attacks. If the stage 2 has detected an attack, it will update SDN to drop the SMTP traffic flows and also update the flow based signature (stage 1) for a future signature attack detection. If the SMTP traffic flows passed the stage 2, it will update SDN for legitimate SMTP traffic flows.

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

The goal of the FlowIDS is to protect the SMTP server from the SMTP flood attack that created by mass mailing. Mass mailing comprises of sending various copy sends to a similar email server. SMTP flood attack utilizing mass mailing is likewise generally executed as a DDoS assault by utilizing the utilization of "zombie" botnets. During the SMTP flood attack the most important in our research is to make sure the SMTP server still can process legitimate SMTP Flood and available to the user to use the service. This thesis has addressed the important to study and analysis of legitimate SMTP flow. The study and analysis of legitimate SMTP flow need to be monitor frequently by daily, monthly and yearly until get the pattern of legitimate SMTP flow. This info will feed to our anomaly detection method algorithm. The dataset history and capture traffic flows must be good enough to be analysis using machine learning such as decision tree classification and deep learning algorithm.

Referring to the state of the arts for SMTP Flood attacks [114,115,116, 117, 18, 29, 31]

the thesis has added extra four network bandwidth performance attributes and one earlier detection performance requirement for research gap comparison, which are novel compared to the precent work by Y.E Oktian [57] our work shown improvement on attack mitigations in term of earlier detection, bandwidth consumption and network recover time. The four network bandwidth performance attributes are listed as follows: Source IP, Destination IP, Duration and Traffic flow rate. For earlier detection performance are divided into two parts as follow:

SMTP attack detection efficiency and network recover time.

Thus, this study focused on three significant contributions of mitigation on SMTP Flood attacks using Software Defined Network: 1) Design A new framework for SMTP flood detection and mitigation using FlowIDS in SDN that can learn to detect the SMTP flood attack, during the attack the FlowIDS allowed both legitimate flow (normal flow) and attack flow until the FlowIDS start to the detect the actual attack flow. 2) A method to identify legitimate SMTP flow using Deep Learning and Decision Tree to increase network bandwidth free during SMTP flood attacks on single-site. 3) A method to collaborate and mitigate SMTP flood attack on multi-site using SDN is a method that allows to collaborate and mitigate the SMTP flood attack

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

on SMTP server close to the source of attacks in other site network topology (early mitigation) by update the rules to all SDN controllers within multi-site network.

In chapter 3, an analysis of botnet attack for SMTP server using SDN is proposed. In this study, we present multi-domain SDNs architecture with an integration of Spamhaus server.

We also discussed the method for analysing SMTP Flood attack flows using a decision tree algorithm classification. This study use POX controller and mininet to simulate all the related internet dataset which include with SMTP flood attack flow. The critical in this study to calculate Round Trip Time (RTT), Retransmission Time Out (RTO) and 3 Wayhandshake (WHS) a design method to analysis and calculate has been done. The result show relation between normal flow and anomaly flow with RTT, RTO, 3WHS, Time To Live (TTL) as signature SMTP attack flow. The main objective of this work to propose a multi-site mitigation SMTP flood attack that can be implement near to source attack within our multi-site network architecture. The proposed method allows SDN Controllers to update the Spamhaus server with latest detected SMTP flood attacks flows signatures. It can help to prevent any SMTP flood attack flow from entering others SDN site network.

Chapter 4, investigated the problem on Dossy packet blocking drop behaviour [57]

which related to our research. In previous chapter the result of decision tree classification be used in this chapter. We have presented a framework for anomaly detection and prevention on SMTP traffic flows, namely FlowIDS. The proposed method allows the FlowIDS to update the ONOS controllers with the latest SMTP Flood Attacks signatures. It will prevent any SMTP Flood attack flow from entering others SDN site. We have shown that by the combination of FlowIDS, ONOS and Suricata, these integrated systems have offered better SMTP flood attack detection and prevention compared to standalone Suricata as the main security parameter. In this work, the most important how to improve the detection rate. We use decision tree classification to analysis legitimate flow of SMTP that go thru our network. About 5 inputs feature extractor such as source IP address, destination IP address, flow rate between source IP address to destination IP address, flow rate between destination IP address to source IP address

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

and duration flow between IP source and IP destination. As output of classification is source IP address, the predicated IP address, will be used to find the value in conversation tool Wireshark.

The flow based algorithm construct in our FlowIDS framework need about 5 features such as Flow_rate_min, Flow_rate_max, Duration_max , Duration_min and Flow average of legitimate or normal SMTP flow. By having normal or legitimate SMTP flow, we can filter on this flow only and the rest of flow we can predict as attack flow. The proposed FlowIDS accuracy 35%

better than Dossy method. Our method will learn to detect the SMTP flood attack using SDN based, during the attack the FlowIDS allowed both legitimate flow (normal flow) and attack flow until the FlowIDS start to the detect the actual attack flow. After the FlowIDS learn the attack the network bandwidth starts recover. For the future work, we are planning to integrate the FlowIDS with multi-site of SDN distributed platform to enhance detection and prevention SMTP flood attacks on the Internet.

Chapter 5, based on previous chapter the improvement of FlowIDS framework in term of detection rate need to be improve. In this chapter, we proposed to use deep learning algorithm (Deep Belief Network) to improve our attribute selection. An attribute is some characteristic of a flow or a collection of flows in a given time window T which maybe represented as a numeric or nominal value. In our work 5 attributes be selected to build our detector such as source IP address, destination IP address, flow rate between source IP address to destination IP address, flow rate between destination IP address to source IP address and duration flow between IP source and IP destination. These attributes are then used as part of an attribute vector which captures the characteristics of a single flow for a given time interval. For example, we note that unlike normal peer-to- peer usage, the attacker flow communication may exhibit a more uniform behavior where upon the SMTP queries to initiate communication with SMTP server on the network continuously, resulting in many uniform sized, small packets which continuously occur. Another observation we may make is that for many protocols, the initial exchange of packets when a client joins a network tends to be unique and follows well defined behavior; this knowledge may allow us to assist in classification by capturing the characteristics

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

of the initial packet exchange and carrying this information forward to subsequent time intervals for that flow. For instance, the first packet size attribute is obtained immediately when the initial flow has been established and is carried on to future time windows to assist in classification.

The simulations of the FlowIDS framework were conducted for DT and DL in a single site. The proposed methods to increase network bandwidth free during the attacks by improve the detection rate. The FlowIDS (Deep Learning) improve detection rate 28% better than FlowIDS (Decision Tree). For future work, we will simulate FlowIDS with Deep Learning algorithm in multi-sites.

Chapter 6, investigate the problem using push back [18] method, the mechanism in which routers upstream of the server under attack are asked to start dropping packets to the server under attack all packet will drop. By using the same method , we already propose this method in early chapter 3 .To deploy multi-site controller, distributed ONOS has been introduced to monitor network between multi-site [5], they being able to communicate with 3 ONOS controller and several function such as distributed store can be used to develop our approach; two main function Mastership store which keeps the mapping between each switch to its master, and Network Topology store which describe the network topology in term of links, switch and hosts. The goal of this chapter is to utilize centralize system that provided by FlowIDS framework between SDN to allow the centralize information regarding network behavior according to its needs. In case of multi-site mitigation against anomaly attacks, the sharing information regarding anomaly attacks between sites. By having real time anomaly attacks information in sites A, B & C and this information being centralize and process by FlowIDS, the method allows to collaborate and mitigate the SMTP flood attack on SMTP server close to the source of attacks in other site network topology (early mitigation) by update the rules to all SDN controllers within multi-site network. The result show multi-site mitigation improve about 17 % faster in network bandwidth recovering time compare to single-site mitigation. The critical work for this chapter is to increase the bandwidth free on other sites. It can be done if SMTP attacks are detected at the source attack side at the early stage and the

Kyushu University, Graduate School of Information Science and Electrical Engineering, Department of Advanced Information Technology, PhD Course

mitigation can be deployed faster before the SMTP attacks are spreading to other sides. This system can be tested only up to 3 sites only due to poor design scalability. For future work, we suggest designing scalability for more than 3 sites and above.