Scaling R Programs with Spark
Recent trends in big data analytics indicate the growing need for interactive analysis of large datasets. In response to this trend, a number of academic and commercial systems have been developed to support such use cases. However, data science surveys show that in addition to relational query processing, data scientists often use tools like R to perform more advanced analysis on data. R is particularly popular as it provides support for structured data processing using data frames and includes a number of packages for statistical analysis and visualization.
However, data analysis using R is limited by the amount of memory available on a single machine and further as R is single threaded it is often impractical to use R on large datasets. Prior research has addressed some of these limitations through better I/O support, integration with Hadoop and by designing distributed R runtimes that can be integrated with DBMS engines.
