SBIR Phase I: Decentralized Artificial Intelligence (AI) Computing Operating System for Accessible and Cost-Effective AI

The broader/commercial impact of this Small Business Innovation Research (SBIR) Phase I project lies in enabling wider, cost-effective, and flexible adoption of artificial intelligence (AI) in enterprise and academia, hence making AI more approachable and beneficial to society. In the competitive landscape of AI infrastructure, this project positions itself as a decentralized alternative to established cloud providers. Traditional cloud solutions often impose high costs and restrict users to proprietary infrastructures, posing challenges for smaller organizations and individual researchers looking to experiment and innovate in AI. This project’s model disrupts this by decentralizing access to compute power through a graphics processing unit (GPU) marketplace, where users can tap into affordable, distributed resources, reducing costs by up to 80% compared to centralized platforms. This economic advantage enables a larger audience to take advantage of advanced AI tools without prohibitive costs associated with such an infrastructure. This project can benefit both the AI consumers and AI infrastructure providers. For the AI consumers, this project provides affordable AI hardware at any scale; this project also provides access to the global pool of a variety of GPUs. For infrastructure providers, this project monetizes idle hardware resources and allows the providers to get access to a global market of AI model users. This Small Business Innovation Research (SBIR) Phase I project aims to address the problems of scarce AI-compute resources and low hardware-utilization in traditional data centers. The objective of this project is to create and develop a system framework that aggregates GPU resources at a global scale. The breakthrough technology of this framework comes from harnessing geo-distributed compute resources with long-latency interconnect and hardware heterogeneity, providing a serverless platform for AI training and inference. The principle of this framework is to decompose and re-scale AI workloads in order to address the major technical challenges faced by the decentralization. This framework stands on three technical pillars: (1) latency- and hardware-heterogeneity-aware AI-model partitioning and scheduling which includes self-adaptive parallelism management, separation of decoding and prefill, and balanced tensor-offloading; (2) model re-scaling using speculative inference; and (3) prediction-guided computation verification. The result is a technology that achieves multi-fold higher performance in the decentralized environment than the state-of-the-art and current industry solutions. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.