Aspects of Adapting Data Collection to Intrusion Detection

Licentiate thesis, 2006

The focus of this thesis is on data collection and in particular data collection for intrusion detection purposes. Data collection is the first, and possibly most important activity in the overall intrusion detection process. The result of the detection can never be better than the data on which the detection is based. One of the main problems in this respect is that the amount of data is too large to be readily processed and significant data reduction is needed early on in the detection process. Consequently, I have developed the Manifestation Extraction Tool for Analysis of Logs (METAL). METAL extracts useful log items, manifestations, from collected data while discarding redundant log items. Identifying manifestations for a specific attack is fundamental as the manifestations hold the information that is needed for detecting the attack. The operation of the METAL tool is based on differential analysis between log data captured during attack activity and corresponding normal activity. The tool will not only provide a set of manifestations, but will also provide a significant reduction in data. In an experiment with buffer overflow attacks and data from system call logs, a data reduction rate of 95\% was achieved. The thesis also studies the relationship between data collection mechanism characteristics and log data, i.e. which types of data can be logged by a specific mechanism. This will in turn provide information on which attacks can be detected using data from a certain mechanism. The result is presented in the form of a taxonomy and a classification of a number of data collection mechanisms.