Mesos Updates

[Benjamin Hindman <be...@EECS.Berkeley.EDU>]

Hi All!

I've been rather silently over the past few months focusing on Mesos. In particular, I have been working at Twitter to help get Mesos deployed and used. I'm thrilled to say that Twitter is invested in seeing the project succeed internally and in the open source community!

There has been a bunch of progress over the past few months that I'm happy to report. I thought I would send a quick "state of the union" report on Mesos at Twitter and discuss what I think is necessary to accomplish for our "first" Apache release.

Twitter has three different clusters running Mesos, a "test" cluster, a "non-production" (nonprod) cluster, and a "production" (prod) cluster. The test cluster is where I incubate new versions of Mesos before they get cascaded through nonprod and prod. The nonprod cluster is mostly used for (1) experimental new services that are being developed internally and (2) load tests. And the prod cluster is being used by numerous "streaming" services that perform different tasks based on data that they are ingesting (for example, these services get data off of the internal equivalent of the Twitter "firehose"). Only a few of the services running in prod and non-prod have daemon style "always up" requirements, but the uptimes have been looking great as of late! There are some promising objectives right around for the corner for Mesos at Twitter, and I'm even more excited to report on those once they happen! This includes running Hadoop on Mesos (not the primary reason Twitter was excited about Mesos in the first place), as well as some rather "important" internal Twitter services ... stay tuned! ;)

There is still lots to be done (which I'll discuss briefly below), but that being said I'd love to shoot for our first Apache release date of early June. I'm not sure the exact protocol for this ... 

There are a few upcoming features that I wanted to hold out on for the first release (all of which are being worked on):
(1) Eliminating SWIG as a dependency for the webui (the biggest blocker I've noticed for people downloading and installing/running the system).
(2) Providing task history information.
(3) Handling slave upgrades/failures (without killing the running tasks).
(4) Launching schedulers via the master and persisting task information across failures.
(5) Implementing our resource hints mechanism, which has been renamed to "requests".

Two more things that I'd like to take care of/understand:

(*) What needs to occur when the time comes to offer some contributors roles as committers?

(*) It sounds like Matei has gotten our SVN stuff all setup, so we can bring the code in from Github. I'm still a big fan of providing access to the code via Github however, I think it's a low barrier of entry to get developers to download, read, and play with the code very easily. I'm not sure how other projects do it, but I've been told that some projects share a presence on both Github and Apache SVN?

If you got all the way through this email, thanks! I'm excited to see Mesos take the next steps!

Ben.

P.S. It appears I'm not on mesos-dev@incubator.apache.org ... I guess I need to add myself?

Re: Mesos Updates

[Ted Dunning <te...@gmail.com>]

On Wed, May 11, 2011 at 11:34 AM, Benjamin Hindman
<be...@eecs.berkeley.edu>wrote:

> I've been rather silently over the past few months focusing on Mesos. In
> particular, I have been working at Twitter to help get Mesos deployed and
> used. I'm thrilled to say that Twitter is invested in seeing the project
> succeed internally and in the open source community!
>

That is great.


> There has been a bunch of progress over the past few months that I'm happy
> to report.


Where has this progress been happening?


> .. There is still lots to be done (which I'll discuss briefly below), but
> that being said I'd love to shoot for our first Apache release date of early
> June. I'm not sure the exact protocol for this


The most important thing for a release is community development.


> ...
>
> There are a few upcoming features that I wanted to hold out on for the
> first release (all of which are being worked on):
>

Are these in JIRA?

I see 0 issues updated in the last 30 days.

(*) What needs to occur when the time comes to offer some contributors roles
> as committers?
>

There needs to be more community contact.  The general Apache rule of thumb
is that if it doesn't happen on the mailing list, it didn't happen.

(*) It sounds like Matei has gotten our SVN stuff all setup, so we can bring
> the code in from Github. I'm still a big fan of providing access to the code
> via Github however, I think it's a low barrier of entry to get developers to
> download, read, and play with the code very easily.


Github maintains read-only git mirrors of all apache projects.  Apache also
provides these.  You don't need to do anything.

I'm not sure how other projects do it, but I've been told that some projects
> share a presence on both Github and Apache SVN?
>

All of the git mirrors are currently read-only.  Apache infra is making
progress on allowing full git access for committers.  In the meantime,
git-svn works pretty well.  I typically put long-lived development branches
on github for sharing and commenting.  I rebase against SVN periodically and
push the rebased stuff to github.  Works well.  Periodically, I pull patches
and post them on JIRA so everybody can play too.  When all is ready, I push
the change via git-svn.

P.S. It appears I'm not on mesos-dev@incubator.apache.org ... I guess I need
> to add myself?


Yes.  And things need to happen on the list.