ROBUSTNET: Replicated Optimized Bucket Storage for Network Data

Current log compression solutions can achieve significant compression by preprocessing logs before applying general byte level compression. These solutions, however, fail to capture structural redundancies across computers of the same service class (e.g., workstations based on the same image). Current pcap compression strategies do not scale well when multiple points of observation are used. Even if these compression strategies scale effectively, this will still leave future analyzers with an incomplete picture because computers that emit packets rarely would not be accounted for. As an example, consider, IOT devices that only emit packets during specific events. It is necessary to have a topology map of the entire network as devices IOT devices may not emit packets during the observation window. Such devices are often used as foot holds in an attack campaign.  To have a complete picture, it is necessary to replicate the portions of the network to recreate the exchanges that occurred during the incident. This would give future analyzers access to things like application memory state and network topology information which are normally inaccessible in existing solutions.  Our objective in this effort is to enable future offline analysis of a network's behavior by employing an updated storage and collection strategy. Our approach will leverage repeated structures across both network capture points and computer services classes to reduce redundant information storage. For logs, we will use template based lexical similarity metrics to bucket similar logs entries. For pcaps, packets will be bucketed by flow. These buckets will be used to assess the probability of observing an item and the item's novelty within the bucket. Items that are relatively frequent and very similar to previous observations will be down sampled to reduce space requirements. The filtering thresholds can be tuned to accommodate the available storage.  In addition, network topology information, traffic flow information, and audits of the application/OS configurations will be stored to enable future replication of computer interactions. An API and intuitive user interface will be created to allow users and ML agents to query the NBB solution and get access to logs (stored or replicated), pcaps, and other state information. Projects like Cyber Agents for Security Testing and Learning Environments (CASTLE) present the API to reinforcement learning agents to gather information from historic networks or newly generated networks.