SBIR Phase I: A Collaborative Architecture to Support Large-Scale Exploratory Workflows

This Small Business Innovation Research Phase I project advances the state of the art and builds fundamental knowledge in the area of workflow systems. This research will produce a set of scalable tools (algorithms and software) for effectively managing complex workflows that will allow users to collaboratively, and in a distributed setting, perform large-scale exploratory tasks. Workflows have been traditionally used to automate repetitive tasks. For applications that are exploratory in nature, very little is repeated; change is the norm. As an engineer or scientist generates and evaluates hypotheses about data under study, a series of different, albeit related, workflows are created while a workflow is adjusted in an interactive process. The proposed effort was designed to manage these rapidly-evolving workflows. If successful, it will streamline the creation, execution and sharing of complex visualizations, data mining or other large-scale data analysis applications. The approach provides infrastructure that can be combined with and enhance existing visualization and workflow systems. Some examples of advanced applications include calibrating simulations for hedge funds and for locating oil wells; mining of usage data for Web analytics; manipulating photos in digital imaging software and creating complex visualizations. The infrastructure developed in this effort will enable users to query over the history of previous explorations; retrieve and re-use workflows relevant to a particular task; contribute new workflows; run workflows remotely; and create live documents that are automatically updated as new results are derived. The proposed infrastructure can be integrated with existing workflow and workflow-based systems and it has the potential to greatly reduce the time to insight in data exploration tasks.