You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mesos.apache.org by Joe Stein <jo...@stealth.ly> on 2014/01/21 01:20:08 UTC
Apache Kafka Scheduler & Executor for Apache mesos
Hi, I have started to work on a scheduler and executor for Apache Kafka
http://kafka.apache.org
I wrote up the high level plan for launching Kafka (brokers, producers,
consumers) on Mesos here
https://cwiki.apache.org/confluence/display/KAFKA/Launching+Kafka+with+Apache+Mesos
I haven't done much yet except implementing the API for scheduler and
executor which I am working on here
https://github.com/stealthly/kafka-mesosand as that becomes functional
posting patches to the KAFKA ticket
https://issues.apache.org/jira/browse/KAFKA-1207 (I also have a test
environment in Vagrant running checked in too)
I will be chipping away at this for the next few weeks (pretty busy until
second week of February) and then should have some focused time to get an
initial implementation (just the broker to start) running (unless I can get
it done before then of course).
Thoughts/Comments/Questions/Concerns/Tomatoes/Feedback welcome!
Thanks!
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
Re: Apache Kafka Scheduler & Executor for Apache mesos
Posted by Benjamin Hindman <be...@eecs.berkeley.edu>.
This is awesome Joe, let us know if we can help!
On Mon, Jan 20, 2014 at 5:20 PM, Joe Stein <jo...@stealth.ly> wrote:
> Hi, I have started to work on a scheduler and executor for Apache Kafka
> http://kafka.apache.org
>
> I wrote up the high level plan for launching Kafka (brokers, producers,
> consumers) on Mesos here
>
> https://cwiki.apache.org/confluence/display/KAFKA/Launching+Kafka+with+Apache+Mesos
>
> I haven't done much yet except implementing the API for scheduler and
> executor which I am working on here
> https://github.com/stealthly/kafka-mesosand as that becomes functional
> posting patches to the KAFKA ticket
> https://issues.apache.org/jira/browse/KAFKA-1207 (I also have a test
> environment in Vagrant running checked in too)
>
> I will be chipping away at this for the next few weeks (pretty busy until
> second week of February) and then should have some focused time to get an
> initial implementation (just the broker to start) running (unless I can get
> it done before then of course).
>
> Thoughts/Comments/Questions/Concerns/Tomatoes/Feedback welcome!
>
> Thanks!
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
Re: Apache Kafka Scheduler & Executor for Apache mesos
Posted by Joe Stein <jo...@stealth.ly>.
from the way the examples where named and the diagram I thought they were
different but realize now the scheduler is the framework :) right?
I will update the kafka wiki page I created to reflect that so not to
confuse others that don't know mesos
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Mon, Jan 20, 2014 at 8:38 PM, Vinod Kone <vi...@gmail.com> wrote:
> Looks great. Thanks for sharing this on the mailing list.
>
> Just curious about your distinction between a framework and scheduler. Are
> they different processes? running on different machines? If so, why the
> distinction?
>
>
> On Mon, Jan 20, 2014 at 4:20 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Hi, I have started to work on a scheduler and executor for Apache Kafka
> > http://kafka.apache.org
> >
> > I wrote up the high level plan for launching Kafka (brokers, producers,
> > consumers) on Mesos here
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Launching+Kafka+with+Apache+Mesos
> >
> > I haven't done much yet except implementing the API for scheduler and
> > executor which I am working on here
> > https://github.com/stealthly/kafka-mesosand as that becomes functional
> > posting patches to the KAFKA ticket
> > https://issues.apache.org/jira/browse/KAFKA-1207 (I also have a test
> > environment in Vagrant running checked in too)
> >
> > I will be chipping away at this for the next few weeks (pretty busy until
> > second week of February) and then should have some focused time to get an
> > initial implementation (just the broker to start) running (unless I can
> get
> > it done before then of course).
> >
> > Thoughts/Comments/Questions/Concerns/Tomatoes/Feedback welcome!
> >
> > Thanks!
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
>
Re: Apache Kafka Scheduler & Executor for Apache mesos
Posted by Vinod Kone <vi...@gmail.com>.
Looks great. Thanks for sharing this on the mailing list.
Just curious about your distinction between a framework and scheduler. Are
they different processes? running on different machines? If so, why the
distinction?
On Mon, Jan 20, 2014 at 4:20 PM, Joe Stein <jo...@stealth.ly> wrote:
> Hi, I have started to work on a scheduler and executor for Apache Kafka
> http://kafka.apache.org
>
> I wrote up the high level plan for launching Kafka (brokers, producers,
> consumers) on Mesos here
>
> https://cwiki.apache.org/confluence/display/KAFKA/Launching+Kafka+with+Apache+Mesos
>
> I haven't done much yet except implementing the API for scheduler and
> executor which I am working on here
> https://github.com/stealthly/kafka-mesosand as that becomes functional
> posting patches to the KAFKA ticket
> https://issues.apache.org/jira/browse/KAFKA-1207 (I also have a test
> environment in Vagrant running checked in too)
>
> I will be chipping away at this for the next few weeks (pretty busy until
> second week of February) and then should have some focused time to get an
> initial implementation (just the broker to start) running (unless I can get
> it done before then of course).
>
> Thoughts/Comments/Questions/Concerns/Tomatoes/Feedback welcome!
>
> Thanks!
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>