Detecting Polymorphic No Operations in Shellcode Based on Mining Techniques

Abstract

Shellcode acts as a weapon to perform Buffer Overflow (BOF), which ranked as the most dangerous vulnerability. It consists of three sections that always transforms its parts to be Polymorphic Shellcode.

Solutions available from Intrusion Detection Systems (IDS) still depend on the signature. Also, solutions that using data mining depends on Shellcode with including payloads and not getting the high results, so polymorphic and unknown Shellcodes could not be detected.

We proposed a new solution using data mining classification technique on special features extracted which depends on operation code of no operation instructions; which can classify the packets on the transport layer of the network as clean or buffer overflow Shellcode attack. This solution can detect unseen Shellcodes.

A dataset generated for malicious packets consist of 500,000 files from Metasploit No-Operation engines and 72,000 files of a clean dataset from various types of data.

By applying different classification methods on the dataset which include selected features we specified and evaluating it by evaluation metrics; show that the solution has achieved high accuracy results with rate 94%. In contrast of signature based on SNORT IDS detects only 50.02% of polymorphic Shellcodes in the experiment that occurred to compare the proposed solution with real IDS system. SVM algorithm selected because of the recall rate 99.33% in detecting polymorphic NOOP’s with low false alarm.