You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@incubator.apache.org by Matei Zaharia <ma...@apache.org> on 2010/12/13 23:08:41 UTC
[PROPOSAL] Mesos Project
We would like to propose Mesos as an incubator proposal.
Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
Thanks,
Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
= Abstract =
Mesos is a cluster manager that provides resource sharing and
isolation across cluster applications.
= Proposal =
Mesos is system for sharing resources between cluster applications such
as Hadoop MapReduce, HBase, MPI, and web applications.
It is motivated by three use cases. First, organizations that use
several of these applications can use Mesos to share nodes between them,
increasing utilization and simplifying management. Second, inspired by
MapReduce, a wide array of new cluster programming frameworks are being
proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
Caffeine. Mesos provides a common interface for such frameworks to share
resources, allowing organizations to use multiple frameworks in the same
cluster. Third, Mesos allows users of a framework such as Hadoop to have
multiple instances of the framework on the same cluster, facilitating
workload isolation and incremental deployment of upgrades.
= Background =
Mesos was inspired by operational issues experienced in large Apache Hadoop
deployments as well as a desire to provide a management system for a
wider range of cluster applications. The Apache Hadoop community has long
realized that the current model of having one instance of MapReduce
control a whole cluster leads to problems with isolation (one job may
cause the master to crash, killing all the other jobs), scalability,
and software upgrades (an upgrade must be deployed on the whole cluster).
Statically partitioning resources into multiple fixed-size MapReduce clusters
is unattractive because it lowers both utilization and data locality.
The community has discussed a two-level scheduling model where a simple,
robust low-level layer enables multiple applications to launch tasks
(https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
with the additional goal of supporting non-Hadoop applications as well.
Mesos started as a research project at UC Berkeley, but is now being
tested at several companies (including Twitter and Facebook), and has attracted
interest from other industry users and researchers as well. We are
therefore proposing to place Mesos in the Apache incubator and build an
open source community around it.
= Rationale =
Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
already exist in the scientific computing community, they are not well
suited for today's data center environment.
These schedulers generally give jobs coarse-grained static allocations of
the cluster (e.g. X nodes for the full duration of the job).
This is problematic because many cluster applications are elastic
(can scale up and down), so utilization is not optimal under static
partitioning, and because data-intensive applications such as MapReduce
need to run a few tasks on every node of the cluster to read data locally.
To address these challenges, Mesos is designed around two principles:
* Fine-grained sharing: Mesos allocates resources at the level of "tasks"
within a job, allowing applications to scale up and down over time and
to take turns accessing data on cluster nodes.
* Application-controlled scheduling: Applications control which nodes
their tasks run on, allowing them to achieve placement goals such as
data locality.
In addition to these principles, Mesos is designed to be simple, scalable
and robust, becuase a cluster manager must be highly available to support
applications and should not become a bottleneck. Application-controlled
scheduling already simplifies our design by pushing much of the complex
logic of tracking job state to applications. In addition, Mesos employs an
optimized C++ message-passing library to achieve scalability and supports
master failover using Apache ZooKeeper.
Mesos already supports running Hadoop and MPI. We plan to add support
for other systems as requested (and contributed) by the community.
= Current Status =
== Meritocracy ==
Our intent with this incubator proposal is to start building a diverse
developer community around Mesos following the Apache meritocracy model.
We have wanted to make the project open source and encourage contributors
from multiple organizations from the start. We plan to provide plenty
of support to new developers and to quickly recruit those who make solid
contributions to committer status.
== Community ==
Mesos is currently being used by developers at Twitter and researchers in
computer science and civil engineering at Berkeley. We hope to extend the user
and developer base further in the future. The current developers and users
are all interested in building a solid open source community around Mesos.
To work towards an open source community, we have been using the GitHub issue
tracker and mailing lists at Berkeley for development discussions within our
group for several months now.
== Core Developers ==
Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
a research project, Mesos was always intended to solve operational issues
with large clusters and to become an open-source project, building on our
successful experience doing research that has been incorporated into Apache Hadoop
(several scheduling algorithms).
== Alignment ==
The ASF is a natural host for Mesos given that it is already the home of
Hadoop, HBase, Cassandra, and other emerging cloud software projects.
Mesos was designed to support Hadoop from the beginning in order to solve
operational challenges in Hadoop clusters, and it aims to support a wide range
of applications beyond Hadoop as well. Mesos complements the existing Apache
cloud computing projects by providing a unified way to manage these systems
and to share resources and data between them.
= Known Risks =
== Orphaned Products ==
With the current core developers of Mesos being graduate students, there
is a risk that these developers will eventually move on to other projects.
However, because of the broad scope of Mesos, we all plan to continue working
on projects related to it in the next several years. We are also actively
working with developers at other organizations, such as Twitter, who are
good candidates to become contributors.
== Inexperience with Open Source ==
All of the core developers are active users and followers of open source.
Matei Zaharia is a Hadoop committer and has experience with the Apache
infrastructure and development process. Andy Konwinski has contributed
patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
has released open source software as part of his PhD work that was adopted
by a Swedish company.
== Homogeneous Developers ==
The current core developers are all researchers (graduate students and a
young professor). However, we hope to establish a developer community
that includes contributors from several corporations, and we are already
working towards this with Twitter and Facebook.
== Reliance on Salaried Developers ==
Given that the project started in an academic research environment, the
core developers are all interested in it primarily for its own sake rather
than for the sake of employment. We all intend to continue working on Mesos
as volunteers.
== Relationships with Other Apache Products ==
Mesos needs to work well with Hadoop, HBase, and other cloud software
projects. Being hosted on the same infrastructure will facilitate this
and ultimately help out both Mesos and the projects that can now be
managed using it. There is, however, a risk that new projects will be built
to run solely on Mesos, introducing a dependency.
== An Excessive Fascination with the Apache Brand ==
While we respect the reputation of the Apache brand and have no doubts that
it will attract contributors and users, our interest is primarily to give
Mesos a solid home as an open source project following an established
development model. Locating the project in Apache will also facilitate
collaboration with Hadoop, HBase, and other Apache cluster computing projects,
as discussed in the Alignment section.
= Documentation =
Information about Mesos can be found at http://mesos.berkeley.edu.
The following sources may be useful to start with:
* Documentation for GitHub release: http://github.com/mesos/mesos/wiki
* Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
* Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
= Initial Source =
Mesos has been under development since spring 2009 by a team of graduate
students and researchers. It is currently hosted on GitHub under a BSD
license at http://github.com/mesos/mesos.
= External Dependencies =
The dependencies all have Apache compatible licenses, including BSD, MIT,
Boost, and Apache 2.0.
= Cryptography =
Not applicable.
= Required Resources =
== Mailing Lists ==
* mesos-private for private PMC discussions (with moderated subscriptions)
* mesos-dev
* mesos-commits
* mesos-user
== Subversion Directory ==
https://svn.apache.org/repos/asf/incubator/mesos
== Issue Tracking ==
JIRA Mesos (MESOS)
== Other Resources ==
The existing code already has unit tests, so we would like a Hudson instance
to run them whenever a new patch is submitted. This can be added after project
creation.
= Initial Committers =
* Ali Ghodsi (ali at sics dot se)
* Benjamin Hindman (benh at eecs dot berkeley dot edu)
* Andy Konwinski (andyk at eecs dot berkeley dot edu)
* Matei Zaharia (matei at apache dot org)
A CLA is already on file for Matei Zaharia.
= Affiliations =
* Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
* Benjamin Hindman (UC Berkeley)
* Andy Konwinski (UC Berkeley)
* Matei Zaharia (UC Berkeley)
= Sponsors =
== Champion ==
Tom White
== Nominated Mentors ==
* Dhruba Borthakur
* Brian McCallister
* Tom White
== Sponsoring Entity ==
Incubator PMC
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Brian McCallister <br...@skife.org>.
+1
-Brian
On Mon, Dec 13, 2010 at 2:08 PM, Matei Zaharia <ma...@apache.org> wrote:
> We would like to propose Mesos as an incubator proposal.
>
> Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
>
> Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
>
> Thanks,
>
> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
>
>
>
> = Abstract =
>
> Mesos is a cluster manager that provides resource sharing and
> isolation across cluster applications.
>
>
>
> = Proposal =
>
> Mesos is system for sharing resources between cluster applications such
> as Hadoop MapReduce, HBase, MPI, and web applications.
> It is motivated by three use cases. First, organizations that use
> several of these applications can use Mesos to share nodes between them,
> increasing utilization and simplifying management. Second, inspired by
> MapReduce, a wide array of new cluster programming frameworks are being
> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
> Caffeine. Mesos provides a common interface for such frameworks to share
> resources, allowing organizations to use multiple frameworks in the same
> cluster. Third, Mesos allows users of a framework such as Hadoop to have
> multiple instances of the framework on the same cluster, facilitating
> workload isolation and incremental deployment of upgrades.
>
>
>
> = Background =
>
> Mesos was inspired by operational issues experienced in large Apache Hadoop
> deployments as well as a desire to provide a management system for a
> wider range of cluster applications. The Apache Hadoop community has long
> realized that the current model of having one instance of MapReduce
> control a whole cluster leads to problems with isolation (one job may
> cause the master to crash, killing all the other jobs), scalability,
> and software upgrades (an upgrade must be deployed on the whole cluster).
> Statically partitioning resources into multiple fixed-size MapReduce clusters
> is unattractive because it lowers both utilization and data locality.
> The community has discussed a two-level scheduling model where a simple,
> robust low-level layer enables multiple applications to launch tasks
> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
> with the additional goal of supporting non-Hadoop applications as well.
>
> Mesos started as a research project at UC Berkeley, but is now being
> tested at several companies (including Twitter and Facebook), and has attracted
> interest from other industry users and researchers as well. We are
> therefore proposing to place Mesos in the Apache incubator and build an
> open source community around it.
>
>
>
> = Rationale =
>
> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
> already exist in the scientific computing community, they are not well
> suited for today's data center environment.
> These schedulers generally give jobs coarse-grained static allocations of
> the cluster (e.g. X nodes for the full duration of the job).
> This is problematic because many cluster applications are elastic
> (can scale up and down), so utilization is not optimal under static
> partitioning, and because data-intensive applications such as MapReduce
> need to run a few tasks on every node of the cluster to read data locally.
> To address these challenges, Mesos is designed around two principles:
>
> * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
> within a job, allowing applications to scale up and down over time and
> to take turns accessing data on cluster nodes.
> * Application-controlled scheduling: Applications control which nodes
> their tasks run on, allowing them to achieve placement goals such as
> data locality.
>
> In addition to these principles, Mesos is designed to be simple, scalable
> and robust, becuase a cluster manager must be highly available to support
> applications and should not become a bottleneck. Application-controlled
> scheduling already simplifies our design by pushing much of the complex
> logic of tracking job state to applications. In addition, Mesos employs an
> optimized C++ message-passing library to achieve scalability and supports
> master failover using Apache ZooKeeper.
>
> Mesos already supports running Hadoop and MPI. We plan to add support
> for other systems as requested (and contributed) by the community.
>
>
>
> = Current Status =
>
> == Meritocracy ==
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Mesos following the Apache meritocracy model.
> We have wanted to make the project open source and encourage contributors
> from multiple organizations from the start. We plan to provide plenty
> of support to new developers and to quickly recruit those who make solid
> contributions to committer status.
>
> == Community ==
>
> Mesos is currently being used by developers at Twitter and researchers in
> computer science and civil engineering at Berkeley. We hope to extend the user
> and developer base further in the future. The current developers and users
> are all interested in building a solid open source community around Mesos.
>
> To work towards an open source community, we have been using the GitHub issue
> tracker and mailing lists at Berkeley for development discussions within our
> group for several months now.
>
> == Core Developers ==
>
> Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
> the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
> a research project, Mesos was always intended to solve operational issues
> with large clusters and to become an open-source project, building on our
> successful experience doing research that has been incorporated into Apache Hadoop
> (several scheduling algorithms).
>
> == Alignment ==
>
> The ASF is a natural host for Mesos given that it is already the home of
> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
> Mesos was designed to support Hadoop from the beginning in order to solve
> operational challenges in Hadoop clusters, and it aims to support a wide range
> of applications beyond Hadoop as well. Mesos complements the existing Apache
> cloud computing projects by providing a unified way to manage these systems
> and to share resources and data between them.
>
>
>
> = Known Risks =
>
> == Orphaned Products ==
>
> With the current core developers of Mesos being graduate students, there
> is a risk that these developers will eventually move on to other projects.
> However, because of the broad scope of Mesos, we all plan to continue working
> on projects related to it in the next several years. We are also actively
> working with developers at other organizations, such as Twitter, who are
> good candidates to become contributors.
>
> == Inexperience with Open Source ==
>
> All of the core developers are active users and followers of open source.
> Matei Zaharia is a Hadoop committer and has experience with the Apache
> infrastructure and development process. Andy Konwinski has contributed
> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
> has released open source software as part of his PhD work that was adopted
> by a Swedish company.
>
> == Homogeneous Developers ==
>
> The current core developers are all researchers (graduate students and a
> young professor). However, we hope to establish a developer community
> that includes contributors from several corporations, and we are already
> working towards this with Twitter and Facebook.
>
> == Reliance on Salaried Developers ==
>
> Given that the project started in an academic research environment, the
> core developers are all interested in it primarily for its own sake rather
> than for the sake of employment. We all intend to continue working on Mesos
> as volunteers.
>
> == Relationships with Other Apache Products ==
>
> Mesos needs to work well with Hadoop, HBase, and other cloud software
> projects. Being hosted on the same infrastructure will facilitate this
> and ultimately help out both Mesos and the projects that can now be
> managed using it. There is, however, a risk that new projects will be built
> to run solely on Mesos, introducing a dependency.
>
> == An Excessive Fascination with the Apache Brand ==
>
> While we respect the reputation of the Apache brand and have no doubts that
> it will attract contributors and users, our interest is primarily to give
> Mesos a solid home as an open source project following an established
> development model. Locating the project in Apache will also facilitate
> collaboration with Hadoop, HBase, and other Apache cluster computing projects,
> as discussed in the Alignment section.
>
>
>
> = Documentation =
>
> Information about Mesos can be found at http://mesos.berkeley.edu.
> The following sources may be useful to start with:
>
> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
> * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
> * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
>
>
>
> = Initial Source =
>
> Mesos has been under development since spring 2009 by a team of graduate
> students and researchers. It is currently hosted on GitHub under a BSD
> license at http://github.com/mesos/mesos.
>
>
>
> = External Dependencies =
>
> The dependencies all have Apache compatible licenses, including BSD, MIT,
> Boost, and Apache 2.0.
>
>
>
> = Cryptography =
>
> Not applicable.
>
>
>
> = Required Resources =
>
> == Mailing Lists ==
>
> * mesos-private for private PMC discussions (with moderated subscriptions)
> * mesos-dev
> * mesos-commits
> * mesos-user
>
>
>
> == Subversion Directory ==
>
> https://svn.apache.org/repos/asf/incubator/mesos
>
>
>
> == Issue Tracking ==
>
> JIRA Mesos (MESOS)
>
>
>
> == Other Resources ==
>
> The existing code already has unit tests, so we would like a Hudson instance
> to run them whenever a new patch is submitted. This can be added after project
> creation.
>
>
>
> = Initial Committers =
>
> * Ali Ghodsi (ali at sics dot se)
> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
> * Matei Zaharia (matei at apache dot org)
>
> A CLA is already on file for Matei Zaharia.
>
>
> = Affiliations =
>
> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
> * Benjamin Hindman (UC Berkeley)
> * Andy Konwinski (UC Berkeley)
> * Matei Zaharia (UC Berkeley)
>
>
>
> = Sponsors =
>
> == Champion ==
>
> Tom White
>
> == Nominated Mentors ==
>
> * Dhruba Borthakur
> * Brian McCallister
> * Tom White
>
> == Sponsoring Entity ==
>
> Incubator PMC
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Fwd: [PROPOSAL] Mesos Project
Posted by Ian Holsman <li...@holsman.net>.
Thought this may be of interest to people on this list.
---------- Forwarded message ----------
From: Matei Zaharia <ma...@apache.org>
Date: Tue, Dec 14, 2010 at 9:08 AM
Subject: [PROPOSAL] Mesos Project
To: general@incubator.apache.org
Cc: Benjamin Hindman <be...@eecs.berkeley.edu>, Andy Konwinski <
andyk@berkeley.edu>, Ali Ghodsi <al...@eecs.berkeley.edu>
We would like to propose Mesos as an incubator proposal.
Mesos is a resource manager for clusters that provides resource sharing and
isolation across distributed applications like Apache Hadoop, MPI, or web
applications. It started as a research project at UC Berkeley, but is now
being used both by other Berkeley groups and at Twitter. We open sourced
Mesos in August and would like to grow a broader community around it.
Our proposal is included below and available on the wiki at
http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing
feedback and questions on the proposal. Also, let us know if you're
interested in being a mentor.
Thanks,
Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
= Abstract =
Mesos is a cluster manager that provides resource sharing and
isolation across cluster applications.
= Proposal =
Mesos is system for sharing resources between cluster applications such
as Hadoop MapReduce, HBase, MPI, and web applications.
It is motivated by three use cases. First, organizations that use
several of these applications can use Mesos to share nodes between them,
increasing utilization and simplifying management. Second, inspired by
MapReduce, a wide array of new cluster programming frameworks are being
proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
Caffeine. Mesos provides a common interface for such frameworks to share
resources, allowing organizations to use multiple frameworks in the same
cluster. Third, Mesos allows users of a framework such as Hadoop to have
multiple instances of the framework on the same cluster, facilitating
workload isolation and incremental deployment of upgrades.
= Background =
Mesos was inspired by operational issues experienced in large Apache Hadoop
deployments as well as a desire to provide a management system for a
wider range of cluster applications. The Apache Hadoop community has long
realized that the current model of having one instance of MapReduce
control a whole cluster leads to problems with isolation (one job may
cause the master to crash, killing all the other jobs), scalability,
and software upgrades (an upgrade must be deployed on the whole cluster).
Statically partitioning resources into multiple fixed-size MapReduce
clusters
is unattractive because it lowers both utilization and data locality.
The community has discussed a two-level scheduling model where a simple,
robust low-level layer enables multiple applications to launch tasks
(https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a
layer,
with the additional goal of supporting non-Hadoop applications as well.
Mesos started as a research project at UC Berkeley, but is now being
tested at several companies (including Twitter and Facebook), and has
attracted
interest from other industry users and researchers as well. We are
therefore proposing to place Mesos in the Apache incubator and build an
open source community around it.
= Rationale =
Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
already exist in the scientific computing community, they are not well
suited for today's data center environment.
These schedulers generally give jobs coarse-grained static allocations of
the cluster (e.g. X nodes for the full duration of the job).
This is problematic because many cluster applications are elastic
(can scale up and down), so utilization is not optimal under static
partitioning, and because data-intensive applications such as MapReduce
need to run a few tasks on every node of the cluster to read data locally.
To address these challenges, Mesos is designed around two principles:
* Fine-grained sharing: Mesos allocates resources at the level of "tasks"
within a job, allowing applications to scale up and down over time and
to take turns accessing data on cluster nodes.
* Application-controlled scheduling: Applications control which nodes
their tasks run on, allowing them to achieve placement goals such as
data locality.
In addition to these principles, Mesos is designed to be simple, scalable
and robust, becuase a cluster manager must be highly available to support
applications and should not become a bottleneck. Application-controlled
scheduling already simplifies our design by pushing much of the complex
logic of tracking job state to applications. In addition, Mesos employs an
optimized C++ message-passing library to achieve scalability and supports
master failover using Apache ZooKeeper.
Mesos already supports running Hadoop and MPI. We plan to add support
for other systems as requested (and contributed) by the community.
= Current Status =
== Meritocracy ==
Our intent with this incubator proposal is to start building a diverse
developer community around Mesos following the Apache meritocracy model.
We have wanted to make the project open source and encourage contributors
from multiple organizations from the start. We plan to provide plenty
of support to new developers and to quickly recruit those who make solid
contributions to committer status.
== Community ==
Mesos is currently being used by developers at Twitter and researchers in
computer science and civil engineering at Berkeley. We hope to extend the
user
and developer base further in the future. The current developers and users
are all interested in building a solid open source community around Mesos.
To work towards an open source community, we have been using the GitHub
issue
tracker and mailing lists at Berkeley for development discussions within our
group for several months now.
== Core Developers ==
Mesos was started by three graduate students at UC Berkeley (Benjamin
Hindman,
Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
a research project, Mesos was always intended to solve operational issues
with large clusters and to become an open-source project, building on our
successful experience doing research that has been incorporated into Apache
Hadoop
(several scheduling algorithms).
== Alignment ==
The ASF is a natural host for Mesos given that it is already the home of
Hadoop, HBase, Cassandra, and other emerging cloud software projects.
Mesos was designed to support Hadoop from the beginning in order to solve
operational challenges in Hadoop clusters, and it aims to support a wide
range
of applications beyond Hadoop as well. Mesos complements the existing Apache
cloud computing projects by providing a unified way to manage these systems
and to share resources and data between them.
= Known Risks =
== Orphaned Products ==
With the current core developers of Mesos being graduate students, there
is a risk that these developers will eventually move on to other projects.
However, because of the broad scope of Mesos, we all plan to continue
working
on projects related to it in the next several years. We are also actively
working with developers at other organizations, such as Twitter, who are
good candidates to become contributors.
== Inexperience with Open Source ==
All of the core developers are active users and followers of open source.
Matei Zaharia is a Hadoop committer and has experience with the Apache
infrastructure and development process. Andy Konwinski has contributed
patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
has released open source software as part of his PhD work that was adopted
by a Swedish company.
== Homogeneous Developers ==
The current core developers are all researchers (graduate students and a
young professor). However, we hope to establish a developer community
that includes contributors from several corporations, and we are already
working towards this with Twitter and Facebook.
== Reliance on Salaried Developers ==
Given that the project started in an academic research environment, the
core developers are all interested in it primarily for its own sake rather
than for the sake of employment. We all intend to continue working on Mesos
as volunteers.
== Relationships with Other Apache Products ==
Mesos needs to work well with Hadoop, HBase, and other cloud software
projects. Being hosted on the same infrastructure will facilitate this
and ultimately help out both Mesos and the projects that can now be
managed using it. There is, however, a risk that new projects will be built
to run solely on Mesos, introducing a dependency.
== An Excessive Fascination with the Apache Brand ==
While we respect the reputation of the Apache brand and have no doubts that
it will attract contributors and users, our interest is primarily to give
Mesos a solid home as an open source project following an established
development model. Locating the project in Apache will also facilitate
collaboration with Hadoop, HBase, and other Apache cluster computing
projects,
as discussed in the Alignment section.
= Documentation =
Information about Mesos can be found at http://mesos.berkeley.edu.
The following sources may be useful to start with:
* Documentation for GitHub release: http://github.com/mesos/mesos/wiki
* Presentation at Hadoop User Group:
http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
* Tech report on system design and current features:
http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI
2011 conference)
= Initial Source =
Mesos has been under development since spring 2009 by a team of graduate
students and researchers. It is currently hosted on GitHub under a BSD
license at http://github.com/mesos/mesos.
= External Dependencies =
The dependencies all have Apache compatible licenses, including BSD, MIT,
Boost, and Apache 2.0.
= Cryptography =
Not applicable.
= Required Resources =
== Mailing Lists ==
* mesos-private for private PMC discussions (with moderated subscriptions)
* mesos-dev
* mesos-commits
* mesos-user
== Subversion Directory ==
https://svn.apache.org/repos/asf/incubator/mesos
== Issue Tracking ==
JIRA Mesos (MESOS)
== Other Resources ==
The existing code already has unit tests, so we would like a Hudson instance
to run them whenever a new patch is submitted. This can be added after
project
creation.
= Initial Committers =
* Ali Ghodsi (ali at sics dot se)
* Benjamin Hindman (benh at eecs dot berkeley dot edu)
* Andy Konwinski (andyk at eecs dot berkeley dot edu)
* Matei Zaharia (matei at apache dot org)
A CLA is already on file for Matei Zaharia.
= Affiliations =
* Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
* Benjamin Hindman (UC Berkeley)
* Andy Konwinski (UC Berkeley)
* Matei Zaharia (UC Berkeley)
= Sponsors =
== Champion ==
Tom White
== Nominated Mentors ==
* Dhruba Borthakur
* Brian McCallister
* Tom White
== Sponsoring Entity ==
Incubator PMC
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Matei Zaharia <ma...@eecs.berkeley.edu>.
Thanks everyone for taking a look at the proposal. If there are no questions, I'll probably start a vote thread tomorrow. We're also glad to take on more mentors at this poit if anyone is interested (we have 3 already, but others are welcome).
Matei
On Dec 16, 2010, at 2:05 AM, Mohammad Nour El-Din <no...@gmail.com> wrote:
> Thanks David for raising this, but IMHO +1 on a proposal is not equal
> to +1 on a vote, for me it means that the proposal is OK and it is
> like a probe of proposal acceptance :).
>
> On Thu, Dec 16, 2010 at 12:54 AM, Tom White <to...@apache.org> wrote:
>> Thanks David. I think we can run a separate vote thread once this
>> proposal thread has run its course. Other recent proposal threads seem
>> to have general votes of support on them. As you say, this phase is a
>> time to respond to any questions that are raised, attract more
>> mentors, and so on.
>>
>> Cheers,
>> Tom
>>
>> On Wed, Dec 15, 2010 at 4:12 PM, David Crossley <cr...@apache.org> wrote:
>>> David Crossley wrote:
>>>> The Proposal phase is for discussing the proposal,
>>>> encouraging people, attracting mentors, ensuring that
>>>> the proposal is ready.
>>>>
>>>> Then the Vote phase is for voting.
>>>
>>> Dunno what to do now that it has started :-)
>>> They gather steam, and we don't want to re-do it.
>>>
>>> Perhaps add "Vote" to the subject line and declare
>>> the end time, then tally everything at the end.
>>>
>>> -David
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
>
>
> --
> Thanks
> - Mohammad Nour
> Author of (WebSphere Application Server Community Edition 2.0 User Guide)
> http://www.redbooks.ibm.com/abstracts/sg247585.html
> - LinkedIn: http://www.linkedin.com/in/mnour
> - Blog: http://tadabborat.blogspot.com
> ----
> "Life is like riding a bicycle. To keep your balance you must keep moving"
> - Albert Einstein
>
> "Writing clean code is what you must do in order to call yourself a
> professional. There is no reasonable excuse for doing anything less
> than your best."
> - Clean Code: A Handbook of Agile Software Craftsmanship
>
> "Stay hungry, stay foolish."
> - Steve Jobs
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Matei Zaharia <ma...@apache.org>.
Thanks everyone for taking a look at the proposal. If there are no questions, I'm going to post a vote thread tomorrow. We're also happy to take on more mentors, so let us know if you'd like to be one.
Matei
On Dec 16, 2010, at 2:05 AM, Mohammad Nour El-Din wrote:
> Thanks David for raising this, but IMHO +1 on a proposal is not equal
> to +1 on a vote, for me it means that the proposal is OK and it is
> like a probe of proposal acceptance :).
>
> On Thu, Dec 16, 2010 at 12:54 AM, Tom White <to...@apache.org> wrote:
>> Thanks David. I think we can run a separate vote thread once this
>> proposal thread has run its course. Other recent proposal threads seem
>> to have general votes of support on them. As you say, this phase is a
>> time to respond to any questions that are raised, attract more
>> mentors, and so on.
>>
>> Cheers,
>> Tom
>>
>> On Wed, Dec 15, 2010 at 4:12 PM, David Crossley <cr...@apache.org> wrote:
>>> David Crossley wrote:
>>>> The Proposal phase is for discussing the proposal,
>>>> encouraging people, attracting mentors, ensuring that
>>>> the proposal is ready.
>>>>
>>>> Then the Vote phase is for voting.
>>>
>>> Dunno what to do now that it has started :-)
>>> They gather steam, and we don't want to re-do it.
>>>
>>> Perhaps add "Vote" to the subject line and declare
>>> the end time, then tally everything at the end.
>>>
>>> -David
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
>
>
> --
> Thanks
> - Mohammad Nour
> Author of (WebSphere Application Server Community Edition 2.0 User Guide)
> http://www.redbooks.ibm.com/abstracts/sg247585.html
> - LinkedIn: http://www.linkedin.com/in/mnour
> - Blog: http://tadabborat.blogspot.com
> ----
> "Life is like riding a bicycle. To keep your balance you must keep moving"
> - Albert Einstein
>
> "Writing clean code is what you must do in order to call yourself a
> professional. There is no reasonable excuse for doing anything less
> than your best."
> - Clean Code: A Handbook of Agile Software Craftsmanship
>
> "Stay hungry, stay foolish."
> - Steve Jobs
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Mohammad Nour El-Din <no...@gmail.com>.
Thanks David for raising this, but IMHO +1 on a proposal is not equal
to +1 on a vote, for me it means that the proposal is OK and it is
like a probe of proposal acceptance :).
On Thu, Dec 16, 2010 at 12:54 AM, Tom White <to...@apache.org> wrote:
> Thanks David. I think we can run a separate vote thread once this
> proposal thread has run its course. Other recent proposal threads seem
> to have general votes of support on them. As you say, this phase is a
> time to respond to any questions that are raised, attract more
> mentors, and so on.
>
> Cheers,
> Tom
>
> On Wed, Dec 15, 2010 at 4:12 PM, David Crossley <cr...@apache.org> wrote:
>> David Crossley wrote:
>>> The Proposal phase is for discussing the proposal,
>>> encouraging people, attracting mentors, ensuring that
>>> the proposal is ready.
>>>
>>> Then the Vote phase is for voting.
>>
>> Dunno what to do now that it has started :-)
>> They gather steam, and we don't want to re-do it.
>>
>> Perhaps add "Vote" to the subject line and declare
>> the end time, then tally everything at the end.
>>
>> -David
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
Thanks
- Mohammad Nour
Author of (WebSphere Application Server Community Edition 2.0 User Guide)
http://www.redbooks.ibm.com/abstracts/sg247585.html
- LinkedIn: http://www.linkedin.com/in/mnour
- Blog: http://tadabborat.blogspot.com
----
"Life is like riding a bicycle. To keep your balance you must keep moving"
- Albert Einstein
"Writing clean code is what you must do in order to call yourself a
professional. There is no reasonable excuse for doing anything less
than your best."
- Clean Code: A Handbook of Agile Software Craftsmanship
"Stay hungry, stay foolish."
- Steve Jobs
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Tom White <to...@apache.org>.
Thanks David. I think we can run a separate vote thread once this
proposal thread has run its course. Other recent proposal threads seem
to have general votes of support on them. As you say, this phase is a
time to respond to any questions that are raised, attract more
mentors, and so on.
Cheers,
Tom
On Wed, Dec 15, 2010 at 4:12 PM, David Crossley <cr...@apache.org> wrote:
> David Crossley wrote:
>> The Proposal phase is for discussing the proposal,
>> encouraging people, attracting mentors, ensuring that
>> the proposal is ready.
>>
>> Then the Vote phase is for voting.
>
> Dunno what to do now that it has started :-)
> They gather steam, and we don't want to re-do it.
>
> Perhaps add "Vote" to the subject line and declare
> the end time, then tally everything at the end.
>
> -David
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by David Crossley <cr...@apache.org>.
David Crossley wrote:
> The Proposal phase is for discussing the proposal,
> encouraging people, attracting mentors, ensuring that
> the proposal is ready.
>
> Then the Vote phase is for voting.
Dunno what to do now that it has started :-)
They gather steam, and we don't want to re-do it.
Perhaps add "Vote" to the subject line and declare
the end time, then tally everything at the end.
-David
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by David Crossley <cr...@apache.org>.
The Proposal phase is for discussing the proposal,
encouraging people, attracting mentors, ensuring that
the proposal is ready.
Then the Vote phase is for voting.
-David
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Tommaso Teofili <to...@gmail.com>.
Very nice proposal!
Tommaso
2010/12/15 Edward J. Yoon <ed...@apache.org>
> Looks very cool +1
>
> On Wed, Dec 15, 2010 at 2:44 PM, Tom White <to...@apache.org> wrote:
> > +1 Mesos will fit in well with other ASF projects like Hadoop and HBase.
> >
> > Tom
> >
> > On Mon, Dec 13, 2010 at 2:08 PM, Matei Zaharia <ma...@apache.org> wrote:
> >> We would like to propose Mesos as an incubator proposal.
> >>
> >> Mesos is a resource manager for clusters that provides resource sharing
> and isolation across distributed applications like Apache Hadoop, MPI, or
> web applications. It started as a research project at UC Berkeley, but is
> now being used both by other Berkeley groups and at Twitter. We open sourced
> Mesos in August and would like to grow a broader community around it.
> >>
> >> Our proposal is included below and available on the wiki at
> http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing
> feedback and questions on the proposal. Also, let us know if you're
> interested in being a mentor.
> >>
> >> Thanks,
> >>
> >> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
> >>
> >>
> >>
> >> = Abstract =
> >>
> >> Mesos is a cluster manager that provides resource sharing and
> >> isolation across cluster applications.
> >>
> >>
> >>
> >> = Proposal =
> >>
> >> Mesos is system for sharing resources between cluster applications such
> >> as Hadoop MapReduce, HBase, MPI, and web applications.
> >> It is motivated by three use cases. First, organizations that use
> >> several of these applications can use Mesos to share nodes between them,
> >> increasing utilization and simplifying management. Second, inspired by
> >> MapReduce, a wide array of new cluster programming frameworks are being
> >> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
> >> Caffeine. Mesos provides a common interface for such frameworks to share
> >> resources, allowing organizations to use multiple frameworks in the same
> >> cluster. Third, Mesos allows users of a framework such as Hadoop to have
> >> multiple instances of the framework on the same cluster, facilitating
> >> workload isolation and incremental deployment of upgrades.
> >>
> >>
> >>
> >> = Background =
> >>
> >> Mesos was inspired by operational issues experienced in large Apache
> Hadoop
> >> deployments as well as a desire to provide a management system for a
> >> wider range of cluster applications. The Apache Hadoop community has
> long
> >> realized that the current model of having one instance of MapReduce
> >> control a whole cluster leads to problems with isolation (one job may
> >> cause the master to crash, killing all the other jobs), scalability,
> >> and software upgrades (an upgrade must be deployed on the whole
> cluster).
> >> Statically partitioning resources into multiple fixed-size MapReduce
> clusters
> >> is unattractive because it lowers both utilization and data locality.
> >> The community has discussed a two-level scheduling model where a simple,
> >> robust low-level layer enables multiple applications to launch tasks
> >> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a
> layer,
> >> with the additional goal of supporting non-Hadoop applications as well.
> >>
> >> Mesos started as a research project at UC Berkeley, but is now being
> >> tested at several companies (including Twitter and Facebook), and has
> attracted
> >> interest from other industry users and researchers as well. We are
> >> therefore proposing to place Mesos in the Apache incubator and build an
> >> open source community around it.
> >>
> >>
> >>
> >> = Rationale =
> >>
> >> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
> >> already exist in the scientific computing community, they are not well
> >> suited for today's data center environment.
> >> These schedulers generally give jobs coarse-grained static allocations
> of
> >> the cluster (e.g. X nodes for the full duration of the job).
> >> This is problematic because many cluster applications are elastic
> >> (can scale up and down), so utilization is not optimal under static
> >> partitioning, and because data-intensive applications such as MapReduce
> >> need to run a few tasks on every node of the cluster to read data
> locally.
> >> To address these challenges, Mesos is designed around two principles:
> >>
> >> * Fine-grained sharing: Mesos allocates resources at the level of
> "tasks"
> >> within a job, allowing applications to scale up and down over time and
> >> to take turns accessing data on cluster nodes.
> >> * Application-controlled scheduling: Applications control which nodes
> >> their tasks run on, allowing them to achieve placement goals such as
> >> data locality.
> >>
> >> In addition to these principles, Mesos is designed to be simple,
> scalable
> >> and robust, becuase a cluster manager must be highly available to
> support
> >> applications and should not become a bottleneck. Application-controlled
> >> scheduling already simplifies our design by pushing much of the complex
> >> logic of tracking job state to applications. In addition, Mesos employs
> an
> >> optimized C++ message-passing library to achieve scalability and
> supports
> >> master failover using Apache ZooKeeper.
> >>
> >> Mesos already supports running Hadoop and MPI. We plan to add support
> >> for other systems as requested (and contributed) by the community.
> >>
> >>
> >>
> >> = Current Status =
> >>
> >> == Meritocracy ==
> >>
> >> Our intent with this incubator proposal is to start building a diverse
> >> developer community around Mesos following the Apache meritocracy model.
> >> We have wanted to make the project open source and encourage
> contributors
> >> from multiple organizations from the start. We plan to provide plenty
> >> of support to new developers and to quickly recruit those who make solid
> >> contributions to committer status.
> >>
> >> == Community ==
> >>
> >> Mesos is currently being used by developers at Twitter and researchers
> in
> >> computer science and civil engineering at Berkeley. We hope to extend
> the user
> >> and developer base further in the future. The current developers and
> users
> >> are all interested in building a solid open source community around
> Mesos.
> >>
> >> To work towards an open source community, we have been using the GitHub
> issue
> >> tracker and mailing lists at Berkeley for development discussions within
> our
> >> group for several months now.
> >>
> >> == Core Developers ==
> >>
> >> Mesos was started by three graduate students at UC Berkeley (Benjamin
> Hindman,
> >> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc
> from
> >> the Swedish Institute of Computer Science (Ali Ghodsi). Although started
> as
> >> a research project, Mesos was always intended to solve operational
> issues
> >> with large clusters and to become an open-source project, building on
> our
> >> successful experience doing research that has been incorporated into
> Apache Hadoop
> >> (several scheduling algorithms).
> >>
> >> == Alignment ==
> >>
> >> The ASF is a natural host for Mesos given that it is already the home of
> >> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
> >> Mesos was designed to support Hadoop from the beginning in order to
> solve
> >> operational challenges in Hadoop clusters, and it aims to support a wide
> range
> >> of applications beyond Hadoop as well. Mesos complements the existing
> Apache
> >> cloud computing projects by providing a unified way to manage these
> systems
> >> and to share resources and data between them.
> >>
> >>
> >>
> >> = Known Risks =
> >>
> >> == Orphaned Products ==
> >>
> >> With the current core developers of Mesos being graduate students, there
> >> is a risk that these developers will eventually move on to other
> projects.
> >> However, because of the broad scope of Mesos, we all plan to continue
> working
> >> on projects related to it in the next several years. We are also
> actively
> >> working with developers at other organizations, such as Twitter, who are
> >> good candidates to become contributors.
> >>
> >> == Inexperience with Open Source ==
> >>
> >> All of the core developers are active users and followers of open
> source.
> >> Matei Zaharia is a Hadoop committer and has experience with the Apache
> >> infrastructure and development process. Andy Konwinski has contributed
> >> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
> >> has released open source software as part of his PhD work that was
> adopted
> >> by a Swedish company.
> >>
> >> == Homogeneous Developers ==
> >>
> >> The current core developers are all researchers (graduate students and a
> >> young professor). However, we hope to establish a developer community
> >> that includes contributors from several corporations, and we are already
> >> working towards this with Twitter and Facebook.
> >>
> >> == Reliance on Salaried Developers ==
> >>
> >> Given that the project started in an academic research environment, the
> >> core developers are all interested in it primarily for its own sake
> rather
> >> than for the sake of employment. We all intend to continue working on
> Mesos
> >> as volunteers.
> >>
> >> == Relationships with Other Apache Products ==
> >>
> >> Mesos needs to work well with Hadoop, HBase, and other cloud software
> >> projects. Being hosted on the same infrastructure will facilitate this
> >> and ultimately help out both Mesos and the projects that can now be
> >> managed using it. There is, however, a risk that new projects will be
> built
> >> to run solely on Mesos, introducing a dependency.
> >>
> >> == An Excessive Fascination with the Apache Brand ==
> >>
> >> While we respect the reputation of the Apache brand and have no doubts
> that
> >> it will attract contributors and users, our interest is primarily to
> give
> >> Mesos a solid home as an open source project following an established
> >> development model. Locating the project in Apache will also facilitate
> >> collaboration with Hadoop, HBase, and other Apache cluster computing
> projects,
> >> as discussed in the Alignment section.
> >>
> >>
> >>
> >> = Documentation =
> >>
> >> Information about Mesos can be found at http://mesos.berkeley.edu.
> >> The following sources may be useful to start with:
> >>
> >> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
> >> * Presentation at Hadoop User Group:
> http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
> >> * Tech report on system design and current features:
> http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI
> 2011 conference)
> >>
> >>
> >>
> >> = Initial Source =
> >>
> >> Mesos has been under development since spring 2009 by a team of graduate
> >> students and researchers. It is currently hosted on GitHub under a BSD
> >> license at http://github.com/mesos/mesos.
> >>
> >>
> >>
> >> = External Dependencies =
> >>
> >> The dependencies all have Apache compatible licenses, including BSD,
> MIT,
> >> Boost, and Apache 2.0.
> >>
> >>
> >>
> >> = Cryptography =
> >>
> >> Not applicable.
> >>
> >>
> >>
> >> = Required Resources =
> >>
> >> == Mailing Lists ==
> >>
> >> * mesos-private for private PMC discussions (with moderated
> subscriptions)
> >> * mesos-dev
> >> * mesos-commits
> >> * mesos-user
> >>
> >>
> >>
> >> == Subversion Directory ==
> >>
> >> https://svn.apache.org/repos/asf/incubator/mesos
> >>
> >>
> >>
> >> == Issue Tracking ==
> >>
> >> JIRA Mesos (MESOS)
> >>
> >>
> >>
> >> == Other Resources ==
> >>
> >> The existing code already has unit tests, so we would like a Hudson
> instance
> >> to run them whenever a new patch is submitted. This can be added after
> project
> >> creation.
> >>
> >>
> >>
> >> = Initial Committers =
> >>
> >> * Ali Ghodsi (ali at sics dot se)
> >> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
> >> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
> >> * Matei Zaharia (matei at apache dot org)
> >>
> >> A CLA is already on file for Matei Zaharia.
> >>
> >>
> >> = Affiliations =
> >>
> >> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
> >> * Benjamin Hindman (UC Berkeley)
> >> * Andy Konwinski (UC Berkeley)
> >> * Matei Zaharia (UC Berkeley)
> >>
> >>
> >>
> >> = Sponsors =
> >>
> >> == Champion ==
> >>
> >> Tom White
> >>
> >> == Nominated Mentors ==
> >>
> >> * Dhruba Borthakur
> >> * Brian McCallister
> >> * Tom White
> >>
> >> == Sponsoring Entity ==
> >>
> >> Incubator PMC
> >>
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> >> For additional commands, e-mail: general-help@incubator.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> > For additional commands, e-mail: general-help@incubator.apache.org
> >
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> edwardyoon@apache.org
> http://blog.udanax.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [PROPOSAL] Mesos Project
Posted by "Edward J. Yoon" <ed...@apache.org>.
Looks very cool +1
On Wed, Dec 15, 2010 at 2:44 PM, Tom White <to...@apache.org> wrote:
> +1 Mesos will fit in well with other ASF projects like Hadoop and HBase.
>
> Tom
>
> On Mon, Dec 13, 2010 at 2:08 PM, Matei Zaharia <ma...@apache.org> wrote:
>> We would like to propose Mesos as an incubator proposal.
>>
>> Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
>>
>> Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
>>
>> Thanks,
>>
>> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
>>
>>
>>
>> = Abstract =
>>
>> Mesos is a cluster manager that provides resource sharing and
>> isolation across cluster applications.
>>
>>
>>
>> = Proposal =
>>
>> Mesos is system for sharing resources between cluster applications such
>> as Hadoop MapReduce, HBase, MPI, and web applications.
>> It is motivated by three use cases. First, organizations that use
>> several of these applications can use Mesos to share nodes between them,
>> increasing utilization and simplifying management. Second, inspired by
>> MapReduce, a wide array of new cluster programming frameworks are being
>> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
>> Caffeine. Mesos provides a common interface for such frameworks to share
>> resources, allowing organizations to use multiple frameworks in the same
>> cluster. Third, Mesos allows users of a framework such as Hadoop to have
>> multiple instances of the framework on the same cluster, facilitating
>> workload isolation and incremental deployment of upgrades.
>>
>>
>>
>> = Background =
>>
>> Mesos was inspired by operational issues experienced in large Apache Hadoop
>> deployments as well as a desire to provide a management system for a
>> wider range of cluster applications. The Apache Hadoop community has long
>> realized that the current model of having one instance of MapReduce
>> control a whole cluster leads to problems with isolation (one job may
>> cause the master to crash, killing all the other jobs), scalability,
>> and software upgrades (an upgrade must be deployed on the whole cluster).
>> Statically partitioning resources into multiple fixed-size MapReduce clusters
>> is unattractive because it lowers both utilization and data locality.
>> The community has discussed a two-level scheduling model where a simple,
>> robust low-level layer enables multiple applications to launch tasks
>> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
>> with the additional goal of supporting non-Hadoop applications as well.
>>
>> Mesos started as a research project at UC Berkeley, but is now being
>> tested at several companies (including Twitter and Facebook), and has attracted
>> interest from other industry users and researchers as well. We are
>> therefore proposing to place Mesos in the Apache incubator and build an
>> open source community around it.
>>
>>
>>
>> = Rationale =
>>
>> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
>> already exist in the scientific computing community, they are not well
>> suited for today's data center environment.
>> These schedulers generally give jobs coarse-grained static allocations of
>> the cluster (e.g. X nodes for the full duration of the job).
>> This is problematic because many cluster applications are elastic
>> (can scale up and down), so utilization is not optimal under static
>> partitioning, and because data-intensive applications such as MapReduce
>> need to run a few tasks on every node of the cluster to read data locally.
>> To address these challenges, Mesos is designed around two principles:
>>
>> * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
>> within a job, allowing applications to scale up and down over time and
>> to take turns accessing data on cluster nodes.
>> * Application-controlled scheduling: Applications control which nodes
>> their tasks run on, allowing them to achieve placement goals such as
>> data locality.
>>
>> In addition to these principles, Mesos is designed to be simple, scalable
>> and robust, becuase a cluster manager must be highly available to support
>> applications and should not become a bottleneck. Application-controlled
>> scheduling already simplifies our design by pushing much of the complex
>> logic of tracking job state to applications. In addition, Mesos employs an
>> optimized C++ message-passing library to achieve scalability and supports
>> master failover using Apache ZooKeeper.
>>
>> Mesos already supports running Hadoop and MPI. We plan to add support
>> for other systems as requested (and contributed) by the community.
>>
>>
>>
>> = Current Status =
>>
>> == Meritocracy ==
>>
>> Our intent with this incubator proposal is to start building a diverse
>> developer community around Mesos following the Apache meritocracy model.
>> We have wanted to make the project open source and encourage contributors
>> from multiple organizations from the start. We plan to provide plenty
>> of support to new developers and to quickly recruit those who make solid
>> contributions to committer status.
>>
>> == Community ==
>>
>> Mesos is currently being used by developers at Twitter and researchers in
>> computer science and civil engineering at Berkeley. We hope to extend the user
>> and developer base further in the future. The current developers and users
>> are all interested in building a solid open source community around Mesos.
>>
>> To work towards an open source community, we have been using the GitHub issue
>> tracker and mailing lists at Berkeley for development discussions within our
>> group for several months now.
>>
>> == Core Developers ==
>>
>> Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
>> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
>> the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
>> a research project, Mesos was always intended to solve operational issues
>> with large clusters and to become an open-source project, building on our
>> successful experience doing research that has been incorporated into Apache Hadoop
>> (several scheduling algorithms).
>>
>> == Alignment ==
>>
>> The ASF is a natural host for Mesos given that it is already the home of
>> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
>> Mesos was designed to support Hadoop from the beginning in order to solve
>> operational challenges in Hadoop clusters, and it aims to support a wide range
>> of applications beyond Hadoop as well. Mesos complements the existing Apache
>> cloud computing projects by providing a unified way to manage these systems
>> and to share resources and data between them.
>>
>>
>>
>> = Known Risks =
>>
>> == Orphaned Products ==
>>
>> With the current core developers of Mesos being graduate students, there
>> is a risk that these developers will eventually move on to other projects.
>> However, because of the broad scope of Mesos, we all plan to continue working
>> on projects related to it in the next several years. We are also actively
>> working with developers at other organizations, such as Twitter, who are
>> good candidates to become contributors.
>>
>> == Inexperience with Open Source ==
>>
>> All of the core developers are active users and followers of open source.
>> Matei Zaharia is a Hadoop committer and has experience with the Apache
>> infrastructure and development process. Andy Konwinski has contributed
>> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
>> has released open source software as part of his PhD work that was adopted
>> by a Swedish company.
>>
>> == Homogeneous Developers ==
>>
>> The current core developers are all researchers (graduate students and a
>> young professor). However, we hope to establish a developer community
>> that includes contributors from several corporations, and we are already
>> working towards this with Twitter and Facebook.
>>
>> == Reliance on Salaried Developers ==
>>
>> Given that the project started in an academic research environment, the
>> core developers are all interested in it primarily for its own sake rather
>> than for the sake of employment. We all intend to continue working on Mesos
>> as volunteers.
>>
>> == Relationships with Other Apache Products ==
>>
>> Mesos needs to work well with Hadoop, HBase, and other cloud software
>> projects. Being hosted on the same infrastructure will facilitate this
>> and ultimately help out both Mesos and the projects that can now be
>> managed using it. There is, however, a risk that new projects will be built
>> to run solely on Mesos, introducing a dependency.
>>
>> == An Excessive Fascination with the Apache Brand ==
>>
>> While we respect the reputation of the Apache brand and have no doubts that
>> it will attract contributors and users, our interest is primarily to give
>> Mesos a solid home as an open source project following an established
>> development model. Locating the project in Apache will also facilitate
>> collaboration with Hadoop, HBase, and other Apache cluster computing projects,
>> as discussed in the Alignment section.
>>
>>
>>
>> = Documentation =
>>
>> Information about Mesos can be found at http://mesos.berkeley.edu.
>> The following sources may be useful to start with:
>>
>> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
>> * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
>> * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
>>
>>
>>
>> = Initial Source =
>>
>> Mesos has been under development since spring 2009 by a team of graduate
>> students and researchers. It is currently hosted on GitHub under a BSD
>> license at http://github.com/mesos/mesos.
>>
>>
>>
>> = External Dependencies =
>>
>> The dependencies all have Apache compatible licenses, including BSD, MIT,
>> Boost, and Apache 2.0.
>>
>>
>>
>> = Cryptography =
>>
>> Not applicable.
>>
>>
>>
>> = Required Resources =
>>
>> == Mailing Lists ==
>>
>> * mesos-private for private PMC discussions (with moderated subscriptions)
>> * mesos-dev
>> * mesos-commits
>> * mesos-user
>>
>>
>>
>> == Subversion Directory ==
>>
>> https://svn.apache.org/repos/asf/incubator/mesos
>>
>>
>>
>> == Issue Tracking ==
>>
>> JIRA Mesos (MESOS)
>>
>>
>>
>> == Other Resources ==
>>
>> The existing code already has unit tests, so we would like a Hudson instance
>> to run them whenever a new patch is submitted. This can be added after project
>> creation.
>>
>>
>>
>> = Initial Committers =
>>
>> * Ali Ghodsi (ali at sics dot se)
>> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
>> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
>> * Matei Zaharia (matei at apache dot org)
>>
>> A CLA is already on file for Matei Zaharia.
>>
>>
>> = Affiliations =
>>
>> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
>> * Benjamin Hindman (UC Berkeley)
>> * Andy Konwinski (UC Berkeley)
>> * Matei Zaharia (UC Berkeley)
>>
>>
>>
>> = Sponsors =
>>
>> == Champion ==
>>
>> Tom White
>>
>> == Nominated Mentors ==
>>
>> * Dhruba Borthakur
>> * Brian McCallister
>> * Tom White
>>
>> == Sponsoring Entity ==
>>
>> Incubator PMC
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
Best Regards, Edward J. Yoon
edwardyoon@apache.org
http://blog.udanax.org
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Mohammad Nour El-Din <no...@gmail.com>.
+1 - IMHO this is a great addition to ASF. Good luck :).
On Wed, Dec 15, 2010 at 5:44 AM, Tom White <to...@apache.org> wrote:
> +1 Mesos will fit in well with other ASF projects like Hadoop and HBase.
>
> Tom
>
> On Mon, Dec 13, 2010 at 2:08 PM, Matei Zaharia <ma...@apache.org> wrote:
>> We would like to propose Mesos as an incubator proposal.
>>
>> Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
>>
>> Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
>>
>> Thanks,
>>
>> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
>>
>>
>>
>> = Abstract =
>>
>> Mesos is a cluster manager that provides resource sharing and
>> isolation across cluster applications.
>>
>>
>>
>> = Proposal =
>>
>> Mesos is system for sharing resources between cluster applications such
>> as Hadoop MapReduce, HBase, MPI, and web applications.
>> It is motivated by three use cases. First, organizations that use
>> several of these applications can use Mesos to share nodes between them,
>> increasing utilization and simplifying management. Second, inspired by
>> MapReduce, a wide array of new cluster programming frameworks are being
>> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
>> Caffeine. Mesos provides a common interface for such frameworks to share
>> resources, allowing organizations to use multiple frameworks in the same
>> cluster. Third, Mesos allows users of a framework such as Hadoop to have
>> multiple instances of the framework on the same cluster, facilitating
>> workload isolation and incremental deployment of upgrades.
>>
>>
>>
>> = Background =
>>
>> Mesos was inspired by operational issues experienced in large Apache Hadoop
>> deployments as well as a desire to provide a management system for a
>> wider range of cluster applications. The Apache Hadoop community has long
>> realized that the current model of having one instance of MapReduce
>> control a whole cluster leads to problems with isolation (one job may
>> cause the master to crash, killing all the other jobs), scalability,
>> and software upgrades (an upgrade must be deployed on the whole cluster).
>> Statically partitioning resources into multiple fixed-size MapReduce clusters
>> is unattractive because it lowers both utilization and data locality.
>> The community has discussed a two-level scheduling model where a simple,
>> robust low-level layer enables multiple applications to launch tasks
>> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
>> with the additional goal of supporting non-Hadoop applications as well.
>>
>> Mesos started as a research project at UC Berkeley, but is now being
>> tested at several companies (including Twitter and Facebook), and has attracted
>> interest from other industry users and researchers as well. We are
>> therefore proposing to place Mesos in the Apache incubator and build an
>> open source community around it.
>>
>>
>>
>> = Rationale =
>>
>> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
>> already exist in the scientific computing community, they are not well
>> suited for today's data center environment.
>> These schedulers generally give jobs coarse-grained static allocations of
>> the cluster (e.g. X nodes for the full duration of the job).
>> This is problematic because many cluster applications are elastic
>> (can scale up and down), so utilization is not optimal under static
>> partitioning, and because data-intensive applications such as MapReduce
>> need to run a few tasks on every node of the cluster to read data locally.
>> To address these challenges, Mesos is designed around two principles:
>>
>> * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
>> within a job, allowing applications to scale up and down over time and
>> to take turns accessing data on cluster nodes.
>> * Application-controlled scheduling: Applications control which nodes
>> their tasks run on, allowing them to achieve placement goals such as
>> data locality.
>>
>> In addition to these principles, Mesos is designed to be simple, scalable
>> and robust, becuase a cluster manager must be highly available to support
>> applications and should not become a bottleneck. Application-controlled
>> scheduling already simplifies our design by pushing much of the complex
>> logic of tracking job state to applications. In addition, Mesos employs an
>> optimized C++ message-passing library to achieve scalability and supports
>> master failover using Apache ZooKeeper.
>>
>> Mesos already supports running Hadoop and MPI. We plan to add support
>> for other systems as requested (and contributed) by the community.
>>
>>
>>
>> = Current Status =
>>
>> == Meritocracy ==
>>
>> Our intent with this incubator proposal is to start building a diverse
>> developer community around Mesos following the Apache meritocracy model.
>> We have wanted to make the project open source and encourage contributors
>> from multiple organizations from the start. We plan to provide plenty
>> of support to new developers and to quickly recruit those who make solid
>> contributions to committer status.
>>
>> == Community ==
>>
>> Mesos is currently being used by developers at Twitter and researchers in
>> computer science and civil engineering at Berkeley. We hope to extend the user
>> and developer base further in the future. The current developers and users
>> are all interested in building a solid open source community around Mesos.
>>
>> To work towards an open source community, we have been using the GitHub issue
>> tracker and mailing lists at Berkeley for development discussions within our
>> group for several months now.
>>
>> == Core Developers ==
>>
>> Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
>> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
>> the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
>> a research project, Mesos was always intended to solve operational issues
>> with large clusters and to become an open-source project, building on our
>> successful experience doing research that has been incorporated into Apache Hadoop
>> (several scheduling algorithms).
>>
>> == Alignment ==
>>
>> The ASF is a natural host for Mesos given that it is already the home of
>> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
>> Mesos was designed to support Hadoop from the beginning in order to solve
>> operational challenges in Hadoop clusters, and it aims to support a wide range
>> of applications beyond Hadoop as well. Mesos complements the existing Apache
>> cloud computing projects by providing a unified way to manage these systems
>> and to share resources and data between them.
>>
>>
>>
>> = Known Risks =
>>
>> == Orphaned Products ==
>>
>> With the current core developers of Mesos being graduate students, there
>> is a risk that these developers will eventually move on to other projects.
>> However, because of the broad scope of Mesos, we all plan to continue working
>> on projects related to it in the next several years. We are also actively
>> working with developers at other organizations, such as Twitter, who are
>> good candidates to become contributors.
>>
>> == Inexperience with Open Source ==
>>
>> All of the core developers are active users and followers of open source.
>> Matei Zaharia is a Hadoop committer and has experience with the Apache
>> infrastructure and development process. Andy Konwinski has contributed
>> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
>> has released open source software as part of his PhD work that was adopted
>> by a Swedish company.
>>
>> == Homogeneous Developers ==
>>
>> The current core developers are all researchers (graduate students and a
>> young professor). However, we hope to establish a developer community
>> that includes contributors from several corporations, and we are already
>> working towards this with Twitter and Facebook.
>>
>> == Reliance on Salaried Developers ==
>>
>> Given that the project started in an academic research environment, the
>> core developers are all interested in it primarily for its own sake rather
>> than for the sake of employment. We all intend to continue working on Mesos
>> as volunteers.
>>
>> == Relationships with Other Apache Products ==
>>
>> Mesos needs to work well with Hadoop, HBase, and other cloud software
>> projects. Being hosted on the same infrastructure will facilitate this
>> and ultimately help out both Mesos and the projects that can now be
>> managed using it. There is, however, a risk that new projects will be built
>> to run solely on Mesos, introducing a dependency.
>>
>> == An Excessive Fascination with the Apache Brand ==
>>
>> While we respect the reputation of the Apache brand and have no doubts that
>> it will attract contributors and users, our interest is primarily to give
>> Mesos a solid home as an open source project following an established
>> development model. Locating the project in Apache will also facilitate
>> collaboration with Hadoop, HBase, and other Apache cluster computing projects,
>> as discussed in the Alignment section.
>>
>>
>>
>> = Documentation =
>>
>> Information about Mesos can be found at http://mesos.berkeley.edu.
>> The following sources may be useful to start with:
>>
>> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
>> * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
>> * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
>>
>>
>>
>> = Initial Source =
>>
>> Mesos has been under development since spring 2009 by a team of graduate
>> students and researchers. It is currently hosted on GitHub under a BSD
>> license at http://github.com/mesos/mesos.
>>
>>
>>
>> = External Dependencies =
>>
>> The dependencies all have Apache compatible licenses, including BSD, MIT,
>> Boost, and Apache 2.0.
>>
>>
>>
>> = Cryptography =
>>
>> Not applicable.
>>
>>
>>
>> = Required Resources =
>>
>> == Mailing Lists ==
>>
>> * mesos-private for private PMC discussions (with moderated subscriptions)
>> * mesos-dev
>> * mesos-commits
>> * mesos-user
>>
>>
>>
>> == Subversion Directory ==
>>
>> https://svn.apache.org/repos/asf/incubator/mesos
>>
>>
>>
>> == Issue Tracking ==
>>
>> JIRA Mesos (MESOS)
>>
>>
>>
>> == Other Resources ==
>>
>> The existing code already has unit tests, so we would like a Hudson instance
>> to run them whenever a new patch is submitted. This can be added after project
>> creation.
>>
>>
>>
>> = Initial Committers =
>>
>> * Ali Ghodsi (ali at sics dot se)
>> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
>> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
>> * Matei Zaharia (matei at apache dot org)
>>
>> A CLA is already on file for Matei Zaharia.
>>
>>
>> = Affiliations =
>>
>> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
>> * Benjamin Hindman (UC Berkeley)
>> * Andy Konwinski (UC Berkeley)
>> * Matei Zaharia (UC Berkeley)
>>
>>
>>
>> = Sponsors =
>>
>> == Champion ==
>>
>> Tom White
>>
>> == Nominated Mentors ==
>>
>> * Dhruba Borthakur
>> * Brian McCallister
>> * Tom White
>>
>> == Sponsoring Entity ==
>>
>> Incubator PMC
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
--
Thanks
- Mohammad Nour
Author of (WebSphere Application Server Community Edition 2.0 User Guide)
http://www.redbooks.ibm.com/abstracts/sg247585.html
- LinkedIn: http://www.linkedin.com/in/mnour
- Blog: http://tadabborat.blogspot.com
----
"Life is like riding a bicycle. To keep your balance you must keep moving"
- Albert Einstein
"Writing clean code is what you must do in order to call yourself a
professional. There is no reasonable excuse for doing anything less
than your best."
- Clean Code: A Handbook of Agile Software Craftsmanship
"Stay hungry, stay foolish."
- Steve Jobs
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Tom White <to...@apache.org>.
+1 Mesos will fit in well with other ASF projects like Hadoop and HBase.
Tom
On Mon, Dec 13, 2010 at 2:08 PM, Matei Zaharia <ma...@apache.org> wrote:
> We would like to propose Mesos as an incubator proposal.
>
> Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
>
> Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
>
> Thanks,
>
> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
>
>
>
> = Abstract =
>
> Mesos is a cluster manager that provides resource sharing and
> isolation across cluster applications.
>
>
>
> = Proposal =
>
> Mesos is system for sharing resources between cluster applications such
> as Hadoop MapReduce, HBase, MPI, and web applications.
> It is motivated by three use cases. First, organizations that use
> several of these applications can use Mesos to share nodes between them,
> increasing utilization and simplifying management. Second, inspired by
> MapReduce, a wide array of new cluster programming frameworks are being
> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
> Caffeine. Mesos provides a common interface for such frameworks to share
> resources, allowing organizations to use multiple frameworks in the same
> cluster. Third, Mesos allows users of a framework such as Hadoop to have
> multiple instances of the framework on the same cluster, facilitating
> workload isolation and incremental deployment of upgrades.
>
>
>
> = Background =
>
> Mesos was inspired by operational issues experienced in large Apache Hadoop
> deployments as well as a desire to provide a management system for a
> wider range of cluster applications. The Apache Hadoop community has long
> realized that the current model of having one instance of MapReduce
> control a whole cluster leads to problems with isolation (one job may
> cause the master to crash, killing all the other jobs), scalability,
> and software upgrades (an upgrade must be deployed on the whole cluster).
> Statically partitioning resources into multiple fixed-size MapReduce clusters
> is unattractive because it lowers both utilization and data locality.
> The community has discussed a two-level scheduling model where a simple,
> robust low-level layer enables multiple applications to launch tasks
> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
> with the additional goal of supporting non-Hadoop applications as well.
>
> Mesos started as a research project at UC Berkeley, but is now being
> tested at several companies (including Twitter and Facebook), and has attracted
> interest from other industry users and researchers as well. We are
> therefore proposing to place Mesos in the Apache incubator and build an
> open source community around it.
>
>
>
> = Rationale =
>
> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
> already exist in the scientific computing community, they are not well
> suited for today's data center environment.
> These schedulers generally give jobs coarse-grained static allocations of
> the cluster (e.g. X nodes for the full duration of the job).
> This is problematic because many cluster applications are elastic
> (can scale up and down), so utilization is not optimal under static
> partitioning, and because data-intensive applications such as MapReduce
> need to run a few tasks on every node of the cluster to read data locally.
> To address these challenges, Mesos is designed around two principles:
>
> * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
> within a job, allowing applications to scale up and down over time and
> to take turns accessing data on cluster nodes.
> * Application-controlled scheduling: Applications control which nodes
> their tasks run on, allowing them to achieve placement goals such as
> data locality.
>
> In addition to these principles, Mesos is designed to be simple, scalable
> and robust, becuase a cluster manager must be highly available to support
> applications and should not become a bottleneck. Application-controlled
> scheduling already simplifies our design by pushing much of the complex
> logic of tracking job state to applications. In addition, Mesos employs an
> optimized C++ message-passing library to achieve scalability and supports
> master failover using Apache ZooKeeper.
>
> Mesos already supports running Hadoop and MPI. We plan to add support
> for other systems as requested (and contributed) by the community.
>
>
>
> = Current Status =
>
> == Meritocracy ==
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Mesos following the Apache meritocracy model.
> We have wanted to make the project open source and encourage contributors
> from multiple organizations from the start. We plan to provide plenty
> of support to new developers and to quickly recruit those who make solid
> contributions to committer status.
>
> == Community ==
>
> Mesos is currently being used by developers at Twitter and researchers in
> computer science and civil engineering at Berkeley. We hope to extend the user
> and developer base further in the future. The current developers and users
> are all interested in building a solid open source community around Mesos.
>
> To work towards an open source community, we have been using the GitHub issue
> tracker and mailing lists at Berkeley for development discussions within our
> group for several months now.
>
> == Core Developers ==
>
> Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
> the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
> a research project, Mesos was always intended to solve operational issues
> with large clusters and to become an open-source project, building on our
> successful experience doing research that has been incorporated into Apache Hadoop
> (several scheduling algorithms).
>
> == Alignment ==
>
> The ASF is a natural host for Mesos given that it is already the home of
> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
> Mesos was designed to support Hadoop from the beginning in order to solve
> operational challenges in Hadoop clusters, and it aims to support a wide range
> of applications beyond Hadoop as well. Mesos complements the existing Apache
> cloud computing projects by providing a unified way to manage these systems
> and to share resources and data between them.
>
>
>
> = Known Risks =
>
> == Orphaned Products ==
>
> With the current core developers of Mesos being graduate students, there
> is a risk that these developers will eventually move on to other projects.
> However, because of the broad scope of Mesos, we all plan to continue working
> on projects related to it in the next several years. We are also actively
> working with developers at other organizations, such as Twitter, who are
> good candidates to become contributors.
>
> == Inexperience with Open Source ==
>
> All of the core developers are active users and followers of open source.
> Matei Zaharia is a Hadoop committer and has experience with the Apache
> infrastructure and development process. Andy Konwinski has contributed
> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
> has released open source software as part of his PhD work that was adopted
> by a Swedish company.
>
> == Homogeneous Developers ==
>
> The current core developers are all researchers (graduate students and a
> young professor). However, we hope to establish a developer community
> that includes contributors from several corporations, and we are already
> working towards this with Twitter and Facebook.
>
> == Reliance on Salaried Developers ==
>
> Given that the project started in an academic research environment, the
> core developers are all interested in it primarily for its own sake rather
> than for the sake of employment. We all intend to continue working on Mesos
> as volunteers.
>
> == Relationships with Other Apache Products ==
>
> Mesos needs to work well with Hadoop, HBase, and other cloud software
> projects. Being hosted on the same infrastructure will facilitate this
> and ultimately help out both Mesos and the projects that can now be
> managed using it. There is, however, a risk that new projects will be built
> to run solely on Mesos, introducing a dependency.
>
> == An Excessive Fascination with the Apache Brand ==
>
> While we respect the reputation of the Apache brand and have no doubts that
> it will attract contributors and users, our interest is primarily to give
> Mesos a solid home as an open source project following an established
> development model. Locating the project in Apache will also facilitate
> collaboration with Hadoop, HBase, and other Apache cluster computing projects,
> as discussed in the Alignment section.
>
>
>
> = Documentation =
>
> Information about Mesos can be found at http://mesos.berkeley.edu.
> The following sources may be useful to start with:
>
> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
> * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
> * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
>
>
>
> = Initial Source =
>
> Mesos has been under development since spring 2009 by a team of graduate
> students and researchers. It is currently hosted on GitHub under a BSD
> license at http://github.com/mesos/mesos.
>
>
>
> = External Dependencies =
>
> The dependencies all have Apache compatible licenses, including BSD, MIT,
> Boost, and Apache 2.0.
>
>
>
> = Cryptography =
>
> Not applicable.
>
>
>
> = Required Resources =
>
> == Mailing Lists ==
>
> * mesos-private for private PMC discussions (with moderated subscriptions)
> * mesos-dev
> * mesos-commits
> * mesos-user
>
>
>
> == Subversion Directory ==
>
> https://svn.apache.org/repos/asf/incubator/mesos
>
>
>
> == Issue Tracking ==
>
> JIRA Mesos (MESOS)
>
>
>
> == Other Resources ==
>
> The existing code already has unit tests, so we would like a Hudson instance
> to run them whenever a new patch is submitted. This can be added after project
> creation.
>
>
>
> = Initial Committers =
>
> * Ali Ghodsi (ali at sics dot se)
> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
> * Matei Zaharia (matei at apache dot org)
>
> A CLA is already on file for Matei Zaharia.
>
>
> = Affiliations =
>
> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
> * Benjamin Hindman (UC Berkeley)
> * Andy Konwinski (UC Berkeley)
> * Matei Zaharia (UC Berkeley)
>
>
>
> = Sponsors =
>
> == Champion ==
>
> Tom White
>
> == Nominated Mentors ==
>
> * Dhruba Borthakur
> * Brian McCallister
> * Tom White
>
> == Sponsoring Entity ==
>
> Incubator PMC
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by "Edward J. Yoon" <ed...@udanax.org>.
FYI,
http://slidesha.re/aLwObu
Sent from my iPhone
On 2010. 12. 14., at 오후 6:24, Tommaso Teofili <to...@gmail.com> wrote:
> I've seen it on genera@ too.
> It seems pretty much interesting and I wonder if we could take advantage
> from integrating such a module in Hama.
> What do you think?
> Cheers,
> Tommaso
Re: [PROPOSAL] Mesos Project
Posted by "Edward J. Yoon" <ed...@udanax.org>.
Well, I think, yes because, the Hama can be used w/ various
distributed systems e.g., HDFS, HBase, ...
Sent from my iPhone
On 2010. 12. 14., at 오후 6:24, Tommaso Teofili <to...@gmail.com> wrote:
> I've seen it on genera@ too.
> It seems pretty much interesting and I wonder if we could take advantage
> from integrating such a module in Hama.
> What do you think?
> Cheers,
> Tommaso
Re: [PROPOSAL] Mesos Project
Posted by Tommaso Teofili <to...@gmail.com>.
I've seen it on genera@ too.
It seems pretty much interesting and I wonder if we could take advantage
from integrating such a module in Hama.
What do you think?
Cheers,
Tommaso
Fwd: [PROPOSAL] Mesos Project
Posted by "Edward J. Yoon" <ed...@udanax.org>.
Sent from my iPhone
Begin forwarded message:
> From: Matei Zaharia <ma...@apache.org>
> Date: 2010년 12월 14일 오전 7시 8분 41초 KST
> To: general@incubator.apache.org
> Cc: Benjamin Hindman <be...@eecs.berkeley.edu>, Andy Konwinski <an...@berkeley.edu>, Ali Ghodsi <al...@eecs.berkeley.edu>
> Subject: [PROPOSAL] Mesos Project
> Reply-To: general@incubator.apache.org
>
> We would like to propose Mesos as an incubator proposal.
>
> Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
>
> Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
>
> Thanks,
>
> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
>
>
>
> = Abstract =
>
> Mesos is a cluster manager that provides resource sharing and
> isolation across cluster applications.
>
>
>
> = Proposal =
>
> Mesos is system for sharing resources between cluster applications such
> as Hadoop MapReduce, HBase, MPI, and web applications.
> It is motivated by three use cases. First, organizations that use
> several of these applications can use Mesos to share nodes between them,
> increasing utilization and simplifying management. Second, inspired by
> MapReduce, a wide array of new cluster programming frameworks are being
> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
> Caffeine. Mesos provides a common interface for such frameworks to share
> resources, allowing organizations to use multiple frameworks in the same
> cluster. Third, Mesos allows users of a framework such as Hadoop to have
> multiple instances of the framework on the same cluster, facilitating
> workload isolation and incremental deployment of upgrades.
>
>
>
> = Background =
>
> Mesos was inspired by operational issues experienced in large Apache Hadoop
> deployments as well as a desire to provide a management system for a
> wider range of cluster applications. The Apache Hadoop community has long
> realized that the current model of having one instance of MapReduce
> control a whole cluster leads to problems with isolation (one job may
> cause the master to crash, killing all the other jobs), scalability,
> and software upgrades (an upgrade must be deployed on the whole cluster).
> Statically partitioning resources into multiple fixed-size MapReduce clusters
> is unattractive because it lowers both utilization and data locality.
> The community has discussed a two-level scheduling model where a simple,
> robust low-level layer enables multiple applications to launch tasks
> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
> with the additional goal of supporting non-Hadoop applications as well.
>
> Mesos started as a research project at UC Berkeley, but is now being
> tested at several companies (including Twitter and Facebook), and has attracted
> interest from other industry users and researchers as well. We are
> therefore proposing to place Mesos in the Apache incubator and build an
> open source community around it.
>
>
>
> = Rationale =
>
> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
> already exist in the scientific computing community, they are not well
> suited for today's data center environment.
> These schedulers generally give jobs coarse-grained static allocations of
> the cluster (e.g. X nodes for the full duration of the job).
> This is problematic because many cluster applications are elastic
> (can scale up and down), so utilization is not optimal under static
> partitioning, and because data-intensive applications such as MapReduce
> need to run a few tasks on every node of the cluster to read data locally.
> To address these challenges, Mesos is designed around two principles:
>
> * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
> within a job, allowing applications to scale up and down over time and
> to take turns accessing data on cluster nodes.
> * Application-controlled scheduling: Applications control which nodes
> their tasks run on, allowing them to achieve placement goals such as
> data locality.
>
> In addition to these principles, Mesos is designed to be simple, scalable
> and robust, becuase a cluster manager must be highly available to support
> applications and should not become a bottleneck. Application-controlled
> scheduling already simplifies our design by pushing much of the complex
> logic of tracking job state to applications. In addition, Mesos employs an
> optimized C++ message-passing library to achieve scalability and supports
> master failover using Apache ZooKeeper.
>
> Mesos already supports running Hadoop and MPI. We plan to add support
> for other systems as requested (and contributed) by the community.
>
>
>
> = Current Status =
>
> == Meritocracy ==
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Mesos following the Apache meritocracy model.
> We have wanted to make the project open source and encourage contributors
> from multiple organizations from the start. We plan to provide plenty
> of support to new developers and to quickly recruit those who make solid
> contributions to committer status.
>
> == Community ==
>
> Mesos is currently being used by developers at Twitter and researchers in
> computer science and civil engineering at Berkeley. We hope to extend the user
> and developer base further in the future. The current developers and users
> are all interested in building a solid open source community around Mesos.
>
> To work towards an open source community, we have been using the GitHub issue
> tracker and mailing lists at Berkeley for development discussions within our
> group for several months now.
>
> == Core Developers ==
>
> Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
> the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
> a research project, Mesos was always intended to solve operational issues
> with large clusters and to become an open-source project, building on our
> successful experience doing research that has been incorporated into Apache Hadoop
> (several scheduling algorithms).
>
> == Alignment ==
>
> The ASF is a natural host for Mesos given that it is already the home of
> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
> Mesos was designed to support Hadoop from the beginning in order to solve
> operational challenges in Hadoop clusters, and it aims to support a wide range
> of applications beyond Hadoop as well. Mesos complements the existing Apache
> cloud computing projects by providing a unified way to manage these systems
> and to share resources and data between them.
>
>
>
> = Known Risks =
>
> == Orphaned Products ==
>
> With the current core developers of Mesos being graduate students, there
> is a risk that these developers will eventually move on to other projects.
> However, because of the broad scope of Mesos, we all plan to continue working
> on projects related to it in the next several years. We are also actively
> working with developers at other organizations, such as Twitter, who are
> good candidates to become contributors.
>
> == Inexperience with Open Source ==
>
> All of the core developers are active users and followers of open source.
> Matei Zaharia is a Hadoop committer and has experience with the Apache
> infrastructure and development process. Andy Konwinski has contributed
> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
> has released open source software as part of his PhD work that was adopted
> by a Swedish company.
>
> == Homogeneous Developers ==
>
> The current core developers are all researchers (graduate students and a
> young professor). However, we hope to establish a developer community
> that includes contributors from several corporations, and we are already
> working towards this with Twitter and Facebook.
>
> == Reliance on Salaried Developers ==
>
> Given that the project started in an academic research environment, the
> core developers are all interested in it primarily for its own sake rather
> than for the sake of employment. We all intend to continue working on Mesos
> as volunteers.
>
> == Relationships with Other Apache Products ==
>
> Mesos needs to work well with Hadoop, HBase, and other cloud software
> projects. Being hosted on the same infrastructure will facilitate this
> and ultimately help out both Mesos and the projects that can now be
> managed using it. There is, however, a risk that new projects will be built
> to run solely on Mesos, introducing a dependency.
>
> == An Excessive Fascination with the Apache Brand ==
>
> While we respect the reputation of the Apache brand and have no doubts that
> it will attract contributors and users, our interest is primarily to give
> Mesos a solid home as an open source project following an established
> development model. Locating the project in Apache will also facilitate
> collaboration with Hadoop, HBase, and other Apache cluster computing projects,
> as discussed in the Alignment section.
>
>
>
> = Documentation =
>
> Information about Mesos can be found at http://mesos.berkeley.edu.
> The following sources may be useful to start with:
>
> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
> * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
> * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
>
>
>
> = Initial Source =
>
> Mesos has been under development since spring 2009 by a team of graduate
> students and researchers. It is currently hosted on GitHub under a BSD
> license at http://github.com/mesos/mesos.
>
>
>
> = External Dependencies =
>
> The dependencies all have Apache compatible licenses, including BSD, MIT,
> Boost, and Apache 2.0.
>
>
>
> = Cryptography =
>
> Not applicable.
>
>
>
> = Required Resources =
>
> == Mailing Lists ==
>
> * mesos-private for private PMC discussions (with moderated subscriptions)
> * mesos-dev
> * mesos-commits
> * mesos-user
>
>
>
> == Subversion Directory ==
>
> https://svn.apache.org/repos/asf/incubator/mesos
>
>
>
> == Issue Tracking ==
>
> JIRA Mesos (MESOS)
>
>
>
> == Other Resources ==
>
> The existing code already has unit tests, so we would like a Hudson instance
> to run them whenever a new patch is submitted. This can be added after project
> creation.
>
>
>
> = Initial Committers =
>
> * Ali Ghodsi (ali at sics dot se)
> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
> * Matei Zaharia (matei at apache dot org)
>
> A CLA is already on file for Matei Zaharia.
>
>
> = Affiliations =
>
> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
> * Benjamin Hindman (UC Berkeley)
> * Andy Konwinski (UC Berkeley)
> * Matei Zaharia (UC Berkeley)
>
>
>
> = Sponsors =
>
> == Champion ==
>
> Tom White
>
> == Nominated Mentors ==
>
> * Dhruba Borthakur
> * Brian McCallister
> * Tom White
>
> == Sponsoring Entity ==
>
> Incubator PMC
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
Re: [PROPOSAL] Mesos Project
Posted by Tim Williams <wi...@gmail.com>.
On Mon, Dec 13, 2010 at 5:08 PM, Matei Zaharia <ma...@apache.org> wrote:
> We would like to propose Mesos as an incubator proposal.
+1, best wishes...
--tim
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by Doug Cutting <cu...@apache.org>.
+1
Doug
On 12/13/2010 02:08 PM, Matei Zaharia wrote:
> We would like to propose Mesos as an incubator proposal.
>
> Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
>
> Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
>
> Thanks,
>
> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
>
>
>
> = Abstract =
>
> Mesos is a cluster manager that provides resource sharing and
> isolation across cluster applications.
>
>
>
> = Proposal =
>
> Mesos is system for sharing resources between cluster applications such
> as Hadoop MapReduce, HBase, MPI, and web applications.
> It is motivated by three use cases. First, organizations that use
> several of these applications can use Mesos to share nodes between them,
> increasing utilization and simplifying management. Second, inspired by
> MapReduce, a wide array of new cluster programming frameworks are being
> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
> Caffeine. Mesos provides a common interface for such frameworks to share
> resources, allowing organizations to use multiple frameworks in the same
> cluster. Third, Mesos allows users of a framework such as Hadoop to have
> multiple instances of the framework on the same cluster, facilitating
> workload isolation and incremental deployment of upgrades.
>
>
>
> = Background =
>
> Mesos was inspired by operational issues experienced in large Apache Hadoop
> deployments as well as a desire to provide a management system for a
> wider range of cluster applications. The Apache Hadoop community has long
> realized that the current model of having one instance of MapReduce
> control a whole cluster leads to problems with isolation (one job may
> cause the master to crash, killing all the other jobs), scalability,
> and software upgrades (an upgrade must be deployed on the whole cluster).
> Statically partitioning resources into multiple fixed-size MapReduce clusters
> is unattractive because it lowers both utilization and data locality.
> The community has discussed a two-level scheduling model where a simple,
> robust low-level layer enables multiple applications to launch tasks
> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
> with the additional goal of supporting non-Hadoop applications as well.
>
> Mesos started as a research project at UC Berkeley, but is now being
> tested at several companies (including Twitter and Facebook), and has attracted
> interest from other industry users and researchers as well. We are
> therefore proposing to place Mesos in the Apache incubator and build an
> open source community around it.
>
>
>
> = Rationale =
>
> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
> already exist in the scientific computing community, they are not well
> suited for today's data center environment.
> These schedulers generally give jobs coarse-grained static allocations of
> the cluster (e.g. X nodes for the full duration of the job).
> This is problematic because many cluster applications are elastic
> (can scale up and down), so utilization is not optimal under static
> partitioning, and because data-intensive applications such as MapReduce
> need to run a few tasks on every node of the cluster to read data locally.
> To address these challenges, Mesos is designed around two principles:
>
> * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
> within a job, allowing applications to scale up and down over time and
> to take turns accessing data on cluster nodes.
> * Application-controlled scheduling: Applications control which nodes
> their tasks run on, allowing them to achieve placement goals such as
> data locality.
>
> In addition to these principles, Mesos is designed to be simple, scalable
> and robust, becuase a cluster manager must be highly available to support
> applications and should not become a bottleneck. Application-controlled
> scheduling already simplifies our design by pushing much of the complex
> logic of tracking job state to applications. In addition, Mesos employs an
> optimized C++ message-passing library to achieve scalability and supports
> master failover using Apache ZooKeeper.
>
> Mesos already supports running Hadoop and MPI. We plan to add support
> for other systems as requested (and contributed) by the community.
>
>
>
> = Current Status =
>
> == Meritocracy ==
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Mesos following the Apache meritocracy model.
> We have wanted to make the project open source and encourage contributors
> from multiple organizations from the start. We plan to provide plenty
> of support to new developers and to quickly recruit those who make solid
> contributions to committer status.
>
> == Community ==
>
> Mesos is currently being used by developers at Twitter and researchers in
> computer science and civil engineering at Berkeley. We hope to extend the user
> and developer base further in the future. The current developers and users
> are all interested in building a solid open source community around Mesos.
>
> To work towards an open source community, we have been using the GitHub issue
> tracker and mailing lists at Berkeley for development discussions within our
> group for several months now.
>
> == Core Developers ==
>
> Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
> the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
> a research project, Mesos was always intended to solve operational issues
> with large clusters and to become an open-source project, building on our
> successful experience doing research that has been incorporated into Apache Hadoop
> (several scheduling algorithms).
>
> == Alignment ==
>
> The ASF is a natural host for Mesos given that it is already the home of
> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
> Mesos was designed to support Hadoop from the beginning in order to solve
> operational challenges in Hadoop clusters, and it aims to support a wide range
> of applications beyond Hadoop as well. Mesos complements the existing Apache
> cloud computing projects by providing a unified way to manage these systems
> and to share resources and data between them.
>
>
>
> = Known Risks =
>
> == Orphaned Products ==
>
> With the current core developers of Mesos being graduate students, there
> is a risk that these developers will eventually move on to other projects.
> However, because of the broad scope of Mesos, we all plan to continue working
> on projects related to it in the next several years. We are also actively
> working with developers at other organizations, such as Twitter, who are
> good candidates to become contributors.
>
> == Inexperience with Open Source ==
>
> All of the core developers are active users and followers of open source.
> Matei Zaharia is a Hadoop committer and has experience with the Apache
> infrastructure and development process. Andy Konwinski has contributed
> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
> has released open source software as part of his PhD work that was adopted
> by a Swedish company.
>
> == Homogeneous Developers ==
>
> The current core developers are all researchers (graduate students and a
> young professor). However, we hope to establish a developer community
> that includes contributors from several corporations, and we are already
> working towards this with Twitter and Facebook.
>
> == Reliance on Salaried Developers ==
>
> Given that the project started in an academic research environment, the
> core developers are all interested in it primarily for its own sake rather
> than for the sake of employment. We all intend to continue working on Mesos
> as volunteers.
>
> == Relationships with Other Apache Products ==
>
> Mesos needs to work well with Hadoop, HBase, and other cloud software
> projects. Being hosted on the same infrastructure will facilitate this
> and ultimately help out both Mesos and the projects that can now be
> managed using it. There is, however, a risk that new projects will be built
> to run solely on Mesos, introducing a dependency.
>
> == An Excessive Fascination with the Apache Brand ==
>
> While we respect the reputation of the Apache brand and have no doubts that
> it will attract contributors and users, our interest is primarily to give
> Mesos a solid home as an open source project following an established
> development model. Locating the project in Apache will also facilitate
> collaboration with Hadoop, HBase, and other Apache cluster computing projects,
> as discussed in the Alignment section.
>
>
>
> = Documentation =
>
> Information about Mesos can be found at http://mesos.berkeley.edu.
> The following sources may be useful to start with:
>
> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
> * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
> * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
>
>
>
> = Initial Source =
>
> Mesos has been under development since spring 2009 by a team of graduate
> students and researchers. It is currently hosted on GitHub under a BSD
> license at http://github.com/mesos/mesos.
>
>
>
> = External Dependencies =
>
> The dependencies all have Apache compatible licenses, including BSD, MIT,
> Boost, and Apache 2.0.
>
>
>
> = Cryptography =
>
> Not applicable.
>
>
>
> = Required Resources =
>
> == Mailing Lists ==
>
> * mesos-private for private PMC discussions (with moderated subscriptions)
> * mesos-dev
> * mesos-commits
> * mesos-user
>
>
>
> == Subversion Directory ==
>
> https://svn.apache.org/repos/asf/incubator/mesos
>
>
>
> == Issue Tracking ==
>
> JIRA Mesos (MESOS)
>
>
>
> == Other Resources ==
>
> The existing code already has unit tests, so we would like a Hudson instance
> to run them whenever a new patch is submitted. This can be added after project
> creation.
>
>
>
> = Initial Committers =
>
> * Ali Ghodsi (ali at sics dot se)
> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
> * Matei Zaharia (matei at apache dot org)
>
> A CLA is already on file for Matei Zaharia.
>
>
> = Affiliations =
>
> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
> * Benjamin Hindman (UC Berkeley)
> * Andy Konwinski (UC Berkeley)
> * Matei Zaharia (UC Berkeley)
>
>
>
> = Sponsors =
>
> == Champion ==
>
> Tom White
>
> == Nominated Mentors ==
>
> * Dhruba Borthakur
> * Brian McCallister
> * Tom White
>
> == Sponsoring Entity ==
>
> Incubator PMC
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1, sounds cool to me!
Cheers,
Chris
On Dec 13, 2010, at 2:08 PM, Matei Zaharia wrote:
> We would like to propose Mesos as an incubator proposal.
>
> Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it.
>
> Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor.
>
> Thanks,
>
> Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi
>
>
>
> = Abstract =
>
> Mesos is a cluster manager that provides resource sharing and
> isolation across cluster applications.
>
>
>
> = Proposal =
>
> Mesos is system for sharing resources between cluster applications such
> as Hadoop MapReduce, HBase, MPI, and web applications.
> It is motivated by three use cases. First, organizations that use
> several of these applications can use Mesos to share nodes between them,
> increasing utilization and simplifying management. Second, inspired by
> MapReduce, a wide array of new cluster programming frameworks are being
> proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and
> Caffeine. Mesos provides a common interface for such frameworks to share
> resources, allowing organizations to use multiple frameworks in the same
> cluster. Third, Mesos allows users of a framework such as Hadoop to have
> multiple instances of the framework on the same cluster, facilitating
> workload isolation and incremental deployment of upgrades.
>
>
>
> = Background =
>
> Mesos was inspired by operational issues experienced in large Apache Hadoop
> deployments as well as a desire to provide a management system for a
> wider range of cluster applications. The Apache Hadoop community has long
> realized that the current model of having one instance of MapReduce
> control a whole cluster leads to problems with isolation (one job may
> cause the master to crash, killing all the other jobs), scalability,
> and software upgrades (an upgrade must be deployed on the whole cluster).
> Statically partitioning resources into multiple fixed-size MapReduce clusters
> is unattractive because it lowers both utilization and data locality.
> The community has discussed a two-level scheduling model where a simple,
> robust low-level layer enables multiple applications to launch tasks
> (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer,
> with the additional goal of supporting non-Hadoop applications as well.
>
> Mesos started as a research project at UC Berkeley, but is now being
> tested at several companies (including Twitter and Facebook), and has attracted
> interest from other industry users and researchers as well. We are
> therefore proposing to place Mesos in the Apache incubator and build an
> open source community around it.
>
>
>
> = Rationale =
>
> Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine)
> already exist in the scientific computing community, they are not well
> suited for today's data center environment.
> These schedulers generally give jobs coarse-grained static allocations of
> the cluster (e.g. X nodes for the full duration of the job).
> This is problematic because many cluster applications are elastic
> (can scale up and down), so utilization is not optimal under static
> partitioning, and because data-intensive applications such as MapReduce
> need to run a few tasks on every node of the cluster to read data locally.
> To address these challenges, Mesos is designed around two principles:
>
> * Fine-grained sharing: Mesos allocates resources at the level of "tasks"
> within a job, allowing applications to scale up and down over time and
> to take turns accessing data on cluster nodes.
> * Application-controlled scheduling: Applications control which nodes
> their tasks run on, allowing them to achieve placement goals such as
> data locality.
>
> In addition to these principles, Mesos is designed to be simple, scalable
> and robust, becuase a cluster manager must be highly available to support
> applications and should not become a bottleneck. Application-controlled
> scheduling already simplifies our design by pushing much of the complex
> logic of tracking job state to applications. In addition, Mesos employs an
> optimized C++ message-passing library to achieve scalability and supports
> master failover using Apache ZooKeeper.
>
> Mesos already supports running Hadoop and MPI. We plan to add support
> for other systems as requested (and contributed) by the community.
>
>
>
> = Current Status =
>
> == Meritocracy ==
>
> Our intent with this incubator proposal is to start building a diverse
> developer community around Mesos following the Apache meritocracy model.
> We have wanted to make the project open source and encourage contributors
> from multiple organizations from the start. We plan to provide plenty
> of support to new developers and to quickly recruit those who make solid
> contributions to committer status.
>
> == Community ==
>
> Mesos is currently being used by developers at Twitter and researchers in
> computer science and civil engineering at Berkeley. We hope to extend the user
> and developer base further in the future. The current developers and users
> are all interested in building a solid open source community around Mesos.
>
> To work towards an open source community, we have been using the GitHub issue
> tracker and mailing lists at Berkeley for development discussions within our
> group for several months now.
>
> == Core Developers ==
>
> Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman,
> Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from
> the Swedish Institute of Computer Science (Ali Ghodsi). Although started as
> a research project, Mesos was always intended to solve operational issues
> with large clusters and to become an open-source project, building on our
> successful experience doing research that has been incorporated into Apache Hadoop
> (several scheduling algorithms).
>
> == Alignment ==
>
> The ASF is a natural host for Mesos given that it is already the home of
> Hadoop, HBase, Cassandra, and other emerging cloud software projects.
> Mesos was designed to support Hadoop from the beginning in order to solve
> operational challenges in Hadoop clusters, and it aims to support a wide range
> of applications beyond Hadoop as well. Mesos complements the existing Apache
> cloud computing projects by providing a unified way to manage these systems
> and to share resources and data between them.
>
>
>
> = Known Risks =
>
> == Orphaned Products ==
>
> With the current core developers of Mesos being graduate students, there
> is a risk that these developers will eventually move on to other projects.
> However, because of the broad scope of Mesos, we all plan to continue working
> on projects related to it in the next several years. We are also actively
> working with developers at other organizations, such as Twitter, who are
> good candidates to become contributors.
>
> == Inexperience with Open Source ==
>
> All of the core developers are active users and followers of open source.
> Matei Zaharia is a Hadoop committer and has experience with the Apache
> infrastructure and development process. Andy Konwinski has contributed
> patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi
> has released open source software as part of his PhD work that was adopted
> by a Swedish company.
>
> == Homogeneous Developers ==
>
> The current core developers are all researchers (graduate students and a
> young professor). However, we hope to establish a developer community
> that includes contributors from several corporations, and we are already
> working towards this with Twitter and Facebook.
>
> == Reliance on Salaried Developers ==
>
> Given that the project started in an academic research environment, the
> core developers are all interested in it primarily for its own sake rather
> than for the sake of employment. We all intend to continue working on Mesos
> as volunteers.
>
> == Relationships with Other Apache Products ==
>
> Mesos needs to work well with Hadoop, HBase, and other cloud software
> projects. Being hosted on the same infrastructure will facilitate this
> and ultimately help out both Mesos and the projects that can now be
> managed using it. There is, however, a risk that new projects will be built
> to run solely on Mesos, introducing a dependency.
>
> == An Excessive Fascination with the Apache Brand ==
>
> While we respect the reputation of the Apache brand and have no doubts that
> it will attract contributors and users, our interest is primarily to give
> Mesos a solid home as an open source project following an established
> development model. Locating the project in Apache will also facilitate
> collaboration with Hadoop, HBase, and other Apache cluster computing projects,
> as discussed in the Alignment section.
>
>
>
> = Documentation =
>
> Information about Mesos can be found at http://mesos.berkeley.edu.
> The following sources may be useful to start with:
>
> * Documentation for GitHub release: http://github.com/mesos/mesos/wiki
> * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf
> * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference)
>
>
>
> = Initial Source =
>
> Mesos has been under development since spring 2009 by a team of graduate
> students and researchers. It is currently hosted on GitHub under a BSD
> license at http://github.com/mesos/mesos.
>
>
>
> = External Dependencies =
>
> The dependencies all have Apache compatible licenses, including BSD, MIT,
> Boost, and Apache 2.0.
>
>
>
> = Cryptography =
>
> Not applicable.
>
>
>
> = Required Resources =
>
> == Mailing Lists ==
>
> * mesos-private for private PMC discussions (with moderated subscriptions)
> * mesos-dev
> * mesos-commits
> * mesos-user
>
>
>
> == Subversion Directory ==
>
> https://svn.apache.org/repos/asf/incubator/mesos
>
>
>
> == Issue Tracking ==
>
> JIRA Mesos (MESOS)
>
>
>
> == Other Resources ==
>
> The existing code already has unit tests, so we would like a Hudson instance
> to run them whenever a new patch is submitted. This can be added after project
> creation.
>
>
>
> = Initial Committers =
>
> * Ali Ghodsi (ali at sics dot se)
> * Benjamin Hindman (benh at eecs dot berkeley dot edu)
> * Andy Konwinski (andyk at eecs dot berkeley dot edu)
> * Matei Zaharia (matei at apache dot org)
>
> A CLA is already on file for Matei Zaharia.
>
>
> = Affiliations =
>
> * Ali Ghodsi (UC Berkeley / Swedish Institute of Computer Science)
> * Benjamin Hindman (UC Berkeley)
> * Andy Konwinski (UC Berkeley)
> * Matei Zaharia (UC Berkeley)
>
>
>
> = Sponsors =
>
> == Champion ==
>
> Tom White
>
> == Nominated Mentors ==
>
> * Dhruba Borthakur
> * Brian McCallister
> * Tom White
>
> == Sponsoring Entity ==
>
> Incubator PMC
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org