PROTEUS (Promoting Transparent Emic Understanding)

Adversaries in the information operations (IO) domain are increasingly distributed, complex, and focused on specific populations.  While recent IO tools help characterize themes, sentiment, and other indicators in messaging, they generally fail to explain or embody audience perspectives with human-like flexibility, subjectivity, and actionable rationale.  Closing this gap with machinery that can assume audience perspectives and express rationale can therefore enhance the IO analysis.  SIFT and the Information Sciences Institute (ISI) at the University of Southern California (USC) propose to develop and validate PROTEUS (Promoting Transparent Emic Understanding) as a multi-modal LLM-based machine assistant for providing audience perspectives on real or hypothetical media and events, with supporting rationale grounded in social science theories and sociocultural statistics. PROTEUS will include a LLM-tuning pipeline to fine-tune its LLM on updated sociocultural data and online media.  Sociocultural data will include international social science instruments such as the World Values Survey, international barometer surveys, and other validated sources, to help PROTEUS' LLM associate target populations to relevant beliefs, values, and attitudes, and these data be updated over time as the world changes.  After being prompted to assume a perspective and respond to real or hypothetical media, PROTEUS' fine-tuned LLM will generate unstructured text, image-based media, and a network of population beliefs/values supporting its differential responses.  These normalized outputs will populate visualizations and drill-downs in PROTEUS' UI, including geospatial heat-maps.  We propose three related Phase 1 evaluations of LLM perspective-taking (metrics: RMSE, R-squared, and text similarity measures) and rationale-generation (metrics: graph distance and F1) to assess tradeoffs and inform Phase 2 directions.  SIFT has identified IO-related transition partners in the U.S. Department of State and DoD, from recent transitions under the DARPA INCAS program, who we will engage starting in Phase 1, in lockstep with our prototyping and validation activities.