Resilient Distributed Datasets: A Fault-Tolerant ...

Resilient Distributed Datasets: A Fault-Tolerant Abstraction forIn-Memory Cluster ComputingMatei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma,Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion StoicaUniversity of California, BerkeleyAbstractWe present Resilient Distributed Datasets (RDDs), a dis-tributed memory abstraction that lets programmers per-form in-memory computations on large clusters in afault-tolerant manner. RDDs are motivated by two typesof applications that current computing frameworks han-dle inefficiently: iterative algorithms and interactive datamining tools. In both cases, keeping data in memorycan improve performance by an order of achieve fault tolerance efficiently, RDDs provide arestricted form of shared memory, based on coarse-grained transformations rather than fine-grained updatesto shared state. However, we show that RDDs are expres-sive enough to capture a wide class of computations, in-cluding recent specialized programming models for iter-ative jobs, such as Pregel, and new applications that thesemodels do not capture.

This section provides an overview of RDDs. We first de-fine RDDs (x2.1) and introduce their programming inter-face in Spark (x2.2). We then compare RDDs with finer-grained shared memory abstractions (x2.3). Finally, we discuss limitations of the RDD model (x2.4). 2.1 RDD Abstraction Formally, an RDD is a read-only, partitioned collection of ...

Loading..

Tags:

  Distributed, Dataset, Resilient, Resilient distributed datasets

Information

Related search queries