Adversarial Resilience Inspection and Security Evaluation (ARISE)

Neural networks have accomplished breakthroughs across a variety of tasks, including object classification, tracking, and reidentification; understanding natural language semantics, parts of speech, and dependency relations; and even problem-solving in dynamic environments. Because neural networks learn in a stochastic manner with an initially random set of parameters, neural networks are fundamentally constrained by the data on which they are trained, and even the best models give the wrong answer a small percentage of the time. Research into the weaknesses of neural networks has identified a variety of adversarial attacks, in which the input to a neural network is manipulated in a way that deliberately causes the network to misbehave, for example by misclassifying an object in an image. Mitigations to adversarial inputs, unfortunately, tend to work only for specific kinds of attacks, and except for the most basic methods, like gradient-based perturbations, there is not a deep understanding of how adversarial attacks affect model behavior or why they are successful. Assured Information Security, Inc. (AIS) proposes Adversarial Resilience Inspection and Security Evaluation (ARISE), an effort to develop a novel model inspection framework to enable granular exploration, visualization, and analysis of neural networks; and to use this framework to characterize the effects of adversarial inputs on the computations of neural networks. The proposed framework will provide a graphical user interface (GUI) where components, operations, and their relationships to each other are represented by an interactive network diagram. These elements would include low-level, primitive operations such as add, multiply, and matrix-multiply, as well as high-level components such as convolutional layers, transformer blocks, and activation functions. Importantly, this capability must be able to work without modification of the source code of the neural network implementation. This ensures that the proposed framework is never limited by the details of a particular model implementation. The proposed framework will also provide the ability to ôprobeö the internal components of the neural network as it is processing an input or batch of inputs. Thus, it will be able to extract the data at any point in the computation and perform analysis and visualization routines relying on low-level, inner-network access. Using the insights gained from characterizing neural network behavior under adversarial inputs, AIS will design adaptive training methods that focus, for example, on individual layers or subsets of neurons to make existing model more resilient. Additionally, AIS will incorporate into existing models a ôrejectionö capability, wherein the model identifies out-of-distribution (OOD) data and refuses to make a prediction. OOD data is data for which a model is unable or unlikely to provide an accurate prediction.