SBIR Phase I: A scale-out data transfer solution for data-intensive enterprises

The broader impact/commercial potential of this Small Business Innovation Research (SBIR) Phase I project will be to enable data-intensive engineering and science organizations to enhance their data transfer speeds without having to administer premature, costly, and disruptive network upgrades. In addition, the findings and results from this project shed a much needed light to the members of data-intensive communities (both commercial and research) regarding the importance of treating data movement as the 4th critical IT dimension, besides the traditional three: storage, compute, and networking. Furthermore, the project's outcome will show that the 4th-dimension must be defined by balanced interactions of the first three to achieve a desired data transfer speed - an important and elusive concept to grasp, but critical in using high-speed digital communications cost-effectively. Compared to the current state-of-the-art, the product resulting from this project will be far more scalable, space and energy efficient, cost-effective, and easy to deploy and use. Finally, the product will demonstrate that it can meet the data transfer needs of major scientific and commercial endeavors of this decade. This should foster accelerated progress in science and engineering, and thus also the overall national economic health. This Small Business Innovation Research (SBIR) Phase I project addresses the need in modern distributed data-intensive engineering and science operations for a 4th IT dimension - data movement - besides storage, compute, and networking. To match the exponential data-growth trend, the product resulting from this project will provide high-performance data transfer and encryption. The design balances and scales in: IOPS, computation power, and network interfaces. It has a cluster-oriented architecture, using peer-to-peer technologies to ease deployment, operation, and usage. Two patent-pending algorithms help tackle data sets containing a mix of small files and very large files, and provide insensitivity to network latency. Its unique optimizations enable effective use of flash storage. The project will develop a simple, effective, and scale-out capable storage tiering software, a scale-out high-speed data encryption approach, and a scale-out approach for implementing more efficient storage I/O intensive applications. It is anticipated that data rates between two existing Data Transfer Nodes (DTNs) front ending parallel data file systems will be comparable with those from two commonly used high performance data transfer applications. With two four-node clusters, the product is anticipated to achieve 155Gbps memory-to-memory over a 16x10Gbps aggregated link and 70Gbps file-to-file with encryption over a 5000 mile 100Gbps link.