Distributed Streams-based Data Mining for Application Intrusion Detection

This research is focused on the application of real-time data mining techniques to the challenge of application layer intrusion detection. Streams-based data mining methods are used to analyze distributed application access patterns and identify abnormalactivity. This data streams approach removes much of the schema complexity and storage burden associated with traditional database implementations and can provide superior computational and storage efficiencies. The data stream model of computation isespecially useful for very large data sets that need to be processed in a single pass. Application requests are examined using a series of clustering analysis perspectives that enable rapid profiling of individual requests against expected patterns. Morecomplex evaluations can be supported by pipelining analysis stages, providing an inherently scalable solution that is well suited to distributed environments. The work proposed here can be applied to both DoD and commercial data analysis challenges-thetechnology has significant potential to improve the performance and flexibility of real-time data mining. The use of centralized data mining architectures may be unsatisfactory for certain applications due to the inherent time delay that accompaniesoff-line processing. It is believed that a streams-based approach will be faster and more scalable than traditional data mining approaches.In the intrusion detection area, traditional perimeter security implementations such as firewalls need to be supplemented with application-level security measures that will limit access to sensitive data once an usual access pattern has been identified.The approach described here may be a viable alternative or supplement to `hardening' each individual application or encrypting data stores. The results of this effort will be applied to application intrusion detection product development targeted at thehigh-end security market.