Electrical Engineering
      and Computer Sciences

Electrical Engineering and Computer Sciences

COLLEGE OF ENGINEERING

UC Berkeley

Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center

Benjamin Hindman, Andrew Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D. Joseph, Randy H. Katz, Scott Shenker and Ion Stoica

EECS Department
University of California, Berkeley
Technical Report No. UCB/EECS-2010-87
May 26, 2010

http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-87.pdf

We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today's frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our experimental results show that Mesos can achieve near-optimal locality when sharing the cluster among diverse frameworks, can scale up to 50,000 nodes, and is resilient to node failures.


BibTeX citation:

@techreport{Hindman:EECS-2010-87,
    Author = {Hindman, Benjamin and Konwinski, Andrew and Zaharia, Matei and Ghodsi, Ali and Joseph, Anthony D. and Katz, Randy H. and Shenker, Scott and Stoica, Ion},
    Title = {Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center},
    Institution = {EECS Department, University of California, Berkeley},
    Year = {2010},
    Month = {May},
    URL = {http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-87.html},
    Number = {UCB/EECS-2010-87},
    Abstract = {We present Mesos, a platform for sharing commodity clusters between multiple diverse cluster computing frameworks, such as Hadoop and MPI. Sharing improves cluster utilization and avoids per-framework data replication. Mesos shares resources in a fine-grained manner, allowing frameworks to achieve data locality by taking turns reading data stored on each machine. To support the sophisticated schedulers of today's frameworks, Mesos introduces a distributed two-level scheduling mechanism called resource offers. Mesos decides how many resources to offer each framework, while frameworks decide which resources to accept and which computations to run on them. Our experimental results show that Mesos can achieve near-optimal locality when sharing the cluster among diverse frameworks, can scale up to 50,000 nodes, and is resilient to node failures.}
}

EndNote citation:

%0 Report
%A Hindman, Benjamin
%A Konwinski, Andrew
%A Zaharia, Matei
%A Ghodsi, Ali
%A Joseph, Anthony D.
%A Katz, Randy H.
%A Shenker, Scott
%A Stoica, Ion
%T Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center
%I EECS Department, University of California, Berkeley
%D 2010
%8 May 26
%@ UCB/EECS-2010-87
%U http://www.eecs.berkeley.edu/Pubs/TechRpts/2010/EECS-2010-87.html
%F Hindman:EECS-2010-87