PRELUDE: Prompt Recording and Engineering for LLM Upgrades and Development Environment

To address the Army’s need for testing LLMs and their prompts, using standard APIs, evaluation pipelines, and prompt discovery, Redshred proposes to develop a new Prompt Recording and Engineering for LLM Upgrades and Development Environment (PRELUDE) system, based on advanced prompt engineering and data tracking methodologies. Specifically, the innovations in classifying prompts, determining prompt similarity or orthogonality, prompt tracking, and other LLMOps tools will enableteh system to create customized LLMs, train and test prompt datasets, and a low-/no-code environment for LLM creation. This involves the integration of MLOps tools to systematically track and classify prompts and data used in fine-tuning LLMs. As a result, this technology offers to meet such specifications by ensuring prompt independence, performing automatic prompt similarity determination, performing systematic tracking and classification of prompts, enhanced performance in creating reliable and portable prompt-based models, support for operations in classified environments, and a low-code/no-code environment for customized LLM development, which directly address the acquisitions program/platform requirements to (1) evaluate prompt performance, (2) track past prompts, (3) track data given to Large Language Models (LLMs), (4) select metrics for LLM evaluation, (5) evaluate LLMs for various tasks (e.g., requirement classification, mixed initiative dialog), (6) split prompts automatically to create training and test prompt datasets, (7) keep prompts from contaminating other prompts (prompt independence), (8) determine automatically prompt similarity or orthogonality, (9) use a standardized API, (10) create customized LLMs, (11) make prompt-based model development more reliable and portable, and (12) provide a low-/no-code environment for making LLMs. In Phase I, RedShred will fulfill all Phase I requirements, demonstrate the feasibility of tracking data and prompts used to create customized LLMs by selecting and processing data through our Redshred platform, generating and classifying prompts, and utilizing LLMOps and MLOps tools for fine-tuning and tracking.