Detection, Analysis and Mitigation of Model Predicated Exploits at Runtime (DAMPER)

Since, ML software and the algorithms that underpin it are complex, and the inner workings of ML systems are often opaque to those who ultimately integrate, test, and evaluate ML functionality, making the determination of “correct” system behavior intractable for human operators. Automated tools are needed to identify emergent issues with models as they are used within contested environments. For example, operators should be made immediately aware when a model produces arbitrary or spurious results after receiving maliciously crafted input. To address these among other concerns, we propose the creation of a tool for Detection, Analysis and Mitigation of Model Predicated Exploits at Runtime (DAMPER). DAMPER will “dampen” spurious behaviors originating in models within ML-powered software and prevent them from damaging the system’s intended functionality. DAMPER will do this preventatively (reducing the likelihood of model-related issues from becoming problematic in the first place), reactively (generating alerts and performing mitigation at machine speed when problems occur), traceably (alerts and remediations will account for the provenance and origin of affected models and issues identified should be tracked holistically within a broader ecosystem of such models), and performantly (without impacting the throughput and intended functionality of a system from the perspective of good users).