SBIR Phase I: Rething Recommendations

This Small Business Innovation Research (SBIR) Phase I project addresses the problem of learning predictive models of individual choice behavior using sparse information on the behavior of any single individual. The intellectual merit of the project is developing a novel parsimonious view of this problem by modeling choice behavior as a distribution over permutations of alternatives, and making this view implementable at scale. A unit of data in this paradigm is a single comparison between two alternatives. Data of this sort can be derived in a variety of contexts ranging from product reviews to transaction data. While being a parsimonious modeling viewpoint, exact computation, or even representing such models is intractable. The project will focus on developing approximate solutions that, in the spirit of recent advances in high-dimensional statistics, exploit the potential of sparse approximations to such models. Given the vast quantities of data available to build such models it will be important for the algorithms developed to be amenable to parallelization in a manner reminiscent of the Map/Reduce computational paradigm. The algorithms developed will fit this paradigm with key algorithmic steps decomposing across data collected for a single individual. In summary, this project will develop a massively parallelizable approach to modeling individual choice behavior using unstructured data from a variety of sources. The broader impact/commercial potential of this project rests in enabling the emerging, all pervasive transition from 'search' to 'discovery'. This transition can be witnessed in sectors ranging from e-commerce to offline retail to matching impressions to advertisers on demand side platforms. The key stumbling block in this transition is the seeming requirement to build attribute rich models for a given context as opposed to a black box approach. The approach taken in this project is of the latter variety. As a concrete example, the task of merchandising requires an offline retailer to decide on the right assortment of products to carry in segments ranging from tooth paste to clothing; the approach here will power such decision making in an entirely data driven fashion. In a different direction, serving ads based on models that capture a surfer's preferences across the various silos of products and topics on the web can be enabled at scale and incredible granularity using the approach here. The level of granularity made possible by the approach here cannot be achieved with 'parametric' attribute driven approaches. In summary, the tools developed in this project have the potential to do for `discovery' what the PageRank algorithm did for search.