You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Joe Stein <jo...@stealth.ly> on 2014/10/18 02:03:17 UTC
Kafka Command Line & Shell
Hi, I have been thinking about the ease of use for operations with Kafka.
We have lots of tools doing a lot of different things and they are all kind
of in different places.
So, what I was thinking is to have a single interface for our tooling
https://issues.apache.org/jira/browse/KAFKA-1694
This would manifest itself in two ways 1) a command line interface 2) a repl
We would have one entry point centrally for all Kafka commands.
kafka <CMD> <ARGS>
kafka createTopic --brokerList etc,
kafka reassignPartition --brokerList etc,
or execute and run the shell
kafka --brokerList localhost
kafka>use topicName;
kafka>set acl='label';
I was thinking that all calls would be initialized through --brokerList and
the broker can tell the KafkaCommandTool what server to connect to for
MetaData.
Thoughts? Tomatoes?
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
Re: Kafka Command Line & Shell
Posted by Colin <co...@clark.ws>.
+100
--
Colin Clark
+1-320-221-9531
> On Oct 17, 2014, at 7:10 PM, Steve Morin <st...@stevemorin.com> wrote:
>
> Joe I think this is great!
>
>> On Fri, Oct 17, 2014 at 5:03 PM, Joe Stein <jo...@stealth.ly> wrote:
>>
>> Hi, I have been thinking about the ease of use for operations with Kafka.
>> We have lots of tools doing a lot of different things and they are all kind
>> of in different places.
>>
>> So, what I was thinking is to have a single interface for our tooling
>> https://issues.apache.org/jira/browse/KAFKA-1694
>>
>> This would manifest itself in two ways 1) a command line interface 2) a
>> repl
>>
>> We would have one entry point centrally for all Kafka commands.
>> kafka <CMD> <ARGS>
>> kafka createTopic --brokerList etc,
>> kafka reassignPartition --brokerList etc,
>>
>> or execute and run the shell
>>
>> kafka --brokerList localhost
>> kafka>use topicName;
>> kafka>set acl='label';
>>
>> I was thinking that all calls would be initialized through --brokerList and
>> the broker can tell the KafkaCommandTool what server to connect to for
>> MetaData.
>>
>> Thoughts? Tomatoes?
>>
>> /*******************************************
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>> ********************************************/
>>
Re: Kafka Command Line & Shell
Posted by Steve Morin <st...@stevemorin.com>.
Joe I think this is great!
On Fri, Oct 17, 2014 at 5:03 PM, Joe Stein <jo...@stealth.ly> wrote:
> Hi, I have been thinking about the ease of use for operations with Kafka.
> We have lots of tools doing a lot of different things and they are all kind
> of in different places.
>
> So, what I was thinking is to have a single interface for our tooling
> https://issues.apache.org/jira/browse/KAFKA-1694
>
> This would manifest itself in two ways 1) a command line interface 2) a
> repl
>
> We would have one entry point centrally for all Kafka commands.
> kafka <CMD> <ARGS>
> kafka createTopic --brokerList etc,
> kafka reassignPartition --brokerList etc,
>
> or execute and run the shell
>
> kafka --brokerList localhost
> kafka>use topicName;
> kafka>set acl='label';
>
> I was thinking that all calls would be initialized through --brokerList and
> the broker can tell the KafkaCommandTool what server to connect to for
> MetaData.
>
> Thoughts? Tomatoes?
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
Re: Kafka Command Line & Shell
Posted by Guozhang Wang <wa...@gmail.com>.
Thanks Joe. I will read the wiki page.
On Tue, Nov 11, 2014 at 11:47 PM, Joe Stein <jo...@stealth.ly> wrote:
> I started writing this up on the wiki
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements
>
> Instead of starting a new thread I figure just continue this one I started.
> I also added another (important) component for centralized management of
> configuration as "global level" much like we have topic level. These
> global configuration would be overridden (perhaps not all) from the
> server.properties on start (so like in case one broker needs a different
> port, sure).
>
> <<One concern I have is that using RQ/RP wire protocol to the
> controller instead of the current way (via ZK admin path) may expose
> concurrency on the admin requests, which may not be supported yet.
>
> Guozhang, take a look at the diagram how I am thinking of this it would be
> a new handle request that will execute the tools pretty much how they are
> today. My thinking is maybe to-do one at a time (so TopicCommand first I
> think) and have what the TopicCommand is doing happen on server and send
> the RQ/RP to the client but execute on the server. If there is something
> not supported we will of course have to deal with that and implement it for
> sure. Once we get one working end to end I think adding the rest will be
> (more or less) concise iterations to get it done. I added your concern to
> the wiki under the "gotchas" section.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
> On Mon, Oct 20, 2014 at 2:15 AM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > One concern I have is that using RQ/RP wire protocol to the controller
> > instead of the current way (via ZK admin path) may expose concurrency on
> > the admin requests, which may not be supported yet.
> >
> > Some initial discussion about this is on KAFKA-1305.
> >
> > Guozhang
> >
> > On Sun, Oct 19, 2014 at 1:55 PM, Joe Stein <jo...@stealth.ly> wrote:
> >
> > > Maybe we should add some AdminMessage RQ/RP wire protocol structure(s)
> > and
> > > let the controller handle it? We could then build the CLI and Shell in
> > the
> > > project both as useful tools and samples for others.
> > >
> > > Making a http interface should be simple after KAFKA-1494 is done which
> > all
> > > client libraries could offer.
> > >
> > > I will update the design tonight/tomorrow and should be able to have
> > > someone starting to work on it this week.
> > >
> > > /*******************************************
> > > Joe Stein
> > > Founder, Principal Consultant
> > > Big Data Open Source Security LLC
> > > http://www.stealth.ly
> > > Twitter: @allthingshadoop
> > > ********************************************/
> > > On Oct 19, 2014 1:21 PM, "Harsha" <ka...@harsha.io> wrote:
> > >
> > > > +1 for Web Api
> > > >
> > > > On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote:
> > > > > Apache Karaf has been doing this for quite a few years, albeit in
> > Java
> > > > > not Scala. Still, their coding approach to creating a CLI probably
> > > > > captures many lessons learned over that time.
> > > > >
> > > > > Glen
> > > > >
> > > > > On 10/17/2014 08:03 PM, Joe Stein wrote:
> > > > > > Hi, I have been thinking about the ease of use for operations
> with
> > > > Kafka.
> > > > > > We have lots of tools doing a lot of different things and they
> are
> > > all
> > > > kind
> > > > > > of in different places.
> > > > > >
> > > > > > So, what I was thinking is to have a single interface for our
> > tooling
> > > > > > https://issues.apache.org/jira/browse/KAFKA-1694
> > > > > >
> > > > > > This would manifest itself in two ways 1) a command line
> interface
> > 2)
> > > > a repl
> > > > > >
> > > > > > We would have one entry point centrally for all Kafka commands.
> > > > > > kafka <CMD> <ARGS>
> > > > > > kafka createTopic --brokerList etc,
> > > > > > kafka reassignPartition --brokerList etc,
> > > > > >
> > > > > > or execute and run the shell
> > > > > >
> > > > > > kafka --brokerList localhost
> > > > > > kafka>use topicName;
> > > > > > kafka>set acl='label';
> > > > > >
> > > > > > I was thinking that all calls would be initialized through
> > > > --brokerList and
> > > > > > the broker can tell the KafkaCommandTool what server to connect
> to
> > > for
> > > > > > MetaData.
> > > > > >
> > > > > > Thoughts? Tomatoes?
> > > > > >
> > > > > > /*******************************************
> > > > > > Joe Stein
> > > > > > Founder, Principal Consultant
> > > > > > Big Data Open Source Security LLC
> > > > > > http://www.stealth.ly
> > > > > > Twitter: @allthingshadoop <
> > http://www.twitter.com/allthingshadoop>
> > > > > > ********************************************/
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
--
-- Guozhang
Re: Kafka Command Line & Shell
Posted by Joe Stein <jo...@stealth.ly>.
I started writing this up on the wiki
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements
Instead of starting a new thread I figure just continue this one I started.
I also added another (important) component for centralized management of
configuration as "global level" much like we have topic level. These
global configuration would be overridden (perhaps not all) from the
server.properties on start (so like in case one broker needs a different
port, sure).
<<One concern I have is that using RQ/RP wire protocol to the
controller instead of the current way (via ZK admin path) may expose
concurrency on the admin requests, which may not be supported yet.
Guozhang, take a look at the diagram how I am thinking of this it would be
a new handle request that will execute the tools pretty much how they are
today. My thinking is maybe to-do one at a time (so TopicCommand first I
think) and have what the TopicCommand is doing happen on server and send
the RQ/RP to the client but execute on the server. If there is something
not supported we will of course have to deal with that and implement it for
sure. Once we get one working end to end I think adding the rest will be
(more or less) concise iterations to get it done. I added your concern to
the wiki under the "gotchas" section.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/
On Mon, Oct 20, 2014 at 2:15 AM, Guozhang Wang <wa...@gmail.com> wrote:
> One concern I have is that using RQ/RP wire protocol to the controller
> instead of the current way (via ZK admin path) may expose concurrency on
> the admin requests, which may not be supported yet.
>
> Some initial discussion about this is on KAFKA-1305.
>
> Guozhang
>
> On Sun, Oct 19, 2014 at 1:55 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> > Maybe we should add some AdminMessage RQ/RP wire protocol structure(s)
> and
> > let the controller handle it? We could then build the CLI and Shell in
> the
> > project both as useful tools and samples for others.
> >
> > Making a http interface should be simple after KAFKA-1494 is done which
> all
> > client libraries could offer.
> >
> > I will update the design tonight/tomorrow and should be able to have
> > someone starting to work on it this week.
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop
> > ********************************************/
> > On Oct 19, 2014 1:21 PM, "Harsha" <ka...@harsha.io> wrote:
> >
> > > +1 for Web Api
> > >
> > > On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote:
> > > > Apache Karaf has been doing this for quite a few years, albeit in
> Java
> > > > not Scala. Still, their coding approach to creating a CLI probably
> > > > captures many lessons learned over that time.
> > > >
> > > > Glen
> > > >
> > > > On 10/17/2014 08:03 PM, Joe Stein wrote:
> > > > > Hi, I have been thinking about the ease of use for operations with
> > > Kafka.
> > > > > We have lots of tools doing a lot of different things and they are
> > all
> > > kind
> > > > > of in different places.
> > > > >
> > > > > So, what I was thinking is to have a single interface for our
> tooling
> > > > > https://issues.apache.org/jira/browse/KAFKA-1694
> > > > >
> > > > > This would manifest itself in two ways 1) a command line interface
> 2)
> > > a repl
> > > > >
> > > > > We would have one entry point centrally for all Kafka commands.
> > > > > kafka <CMD> <ARGS>
> > > > > kafka createTopic --brokerList etc,
> > > > > kafka reassignPartition --brokerList etc,
> > > > >
> > > > > or execute and run the shell
> > > > >
> > > > > kafka --brokerList localhost
> > > > > kafka>use topicName;
> > > > > kafka>set acl='label';
> > > > >
> > > > > I was thinking that all calls would be initialized through
> > > --brokerList and
> > > > > the broker can tell the KafkaCommandTool what server to connect to
> > for
> > > > > MetaData.
> > > > >
> > > > > Thoughts? Tomatoes?
> > > > >
> > > > > /*******************************************
> > > > > Joe Stein
> > > > > Founder, Principal Consultant
> > > > > Big Data Open Source Security LLC
> > > > > http://www.stealth.ly
> > > > > Twitter: @allthingshadoop <
> http://www.twitter.com/allthingshadoop>
> > > > > ********************************************/
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
Re: Kafka Command Line & Shell
Posted by Guozhang Wang <wa...@gmail.com>.
One concern I have is that using RQ/RP wire protocol to the controller
instead of the current way (via ZK admin path) may expose concurrency on
the admin requests, which may not be supported yet.
Some initial discussion about this is on KAFKA-1305.
Guozhang
On Sun, Oct 19, 2014 at 1:55 PM, Joe Stein <jo...@stealth.ly> wrote:
> Maybe we should add some AdminMessage RQ/RP wire protocol structure(s) and
> let the controller handle it? We could then build the CLI and Shell in the
> project both as useful tools and samples for others.
>
> Making a http interface should be simple after KAFKA-1494 is done which all
> client libraries could offer.
>
> I will update the design tonight/tomorrow and should be able to have
> someone starting to work on it this week.
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop
> ********************************************/
> On Oct 19, 2014 1:21 PM, "Harsha" <ka...@harsha.io> wrote:
>
> > +1 for Web Api
> >
> > On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote:
> > > Apache Karaf has been doing this for quite a few years, albeit in Java
> > > not Scala. Still, their coding approach to creating a CLI probably
> > > captures many lessons learned over that time.
> > >
> > > Glen
> > >
> > > On 10/17/2014 08:03 PM, Joe Stein wrote:
> > > > Hi, I have been thinking about the ease of use for operations with
> > Kafka.
> > > > We have lots of tools doing a lot of different things and they are
> all
> > kind
> > > > of in different places.
> > > >
> > > > So, what I was thinking is to have a single interface for our tooling
> > > > https://issues.apache.org/jira/browse/KAFKA-1694
> > > >
> > > > This would manifest itself in two ways 1) a command line interface 2)
> > a repl
> > > >
> > > > We would have one entry point centrally for all Kafka commands.
> > > > kafka <CMD> <ARGS>
> > > > kafka createTopic --brokerList etc,
> > > > kafka reassignPartition --brokerList etc,
> > > >
> > > > or execute and run the shell
> > > >
> > > > kafka --brokerList localhost
> > > > kafka>use topicName;
> > > > kafka>set acl='label';
> > > >
> > > > I was thinking that all calls would be initialized through
> > --brokerList and
> > > > the broker can tell the KafkaCommandTool what server to connect to
> for
> > > > MetaData.
> > > >
> > > > Thoughts? Tomatoes?
> > > >
> > > > /*******************************************
> > > > Joe Stein
> > > > Founder, Principal Consultant
> > > > Big Data Open Source Security LLC
> > > > http://www.stealth.ly
> > > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > > ********************************************/
> > > >
> > >
> >
>
--
-- Guozhang
Re: Kafka Command Line & Shell
Posted by Joe Stein <jo...@stealth.ly>.
Maybe we should add some AdminMessage RQ/RP wire protocol structure(s) and
let the controller handle it? We could then build the CLI and Shell in the
project both as useful tools and samples for others.
Making a http interface should be simple after KAFKA-1494 is done which all
client libraries could offer.
I will update the design tonight/tomorrow and should be able to have
someone starting to work on it this week.
/*******************************************
Joe Stein
Founder, Principal Consultant
Big Data Open Source Security LLC
http://www.stealth.ly
Twitter: @allthingshadoop
********************************************/
On Oct 19, 2014 1:21 PM, "Harsha" <ka...@harsha.io> wrote:
> +1 for Web Api
>
> On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote:
> > Apache Karaf has been doing this for quite a few years, albeit in Java
> > not Scala. Still, their coding approach to creating a CLI probably
> > captures many lessons learned over that time.
> >
> > Glen
> >
> > On 10/17/2014 08:03 PM, Joe Stein wrote:
> > > Hi, I have been thinking about the ease of use for operations with
> Kafka.
> > > We have lots of tools doing a lot of different things and they are all
> kind
> > > of in different places.
> > >
> > > So, what I was thinking is to have a single interface for our tooling
> > > https://issues.apache.org/jira/browse/KAFKA-1694
> > >
> > > This would manifest itself in two ways 1) a command line interface 2)
> a repl
> > >
> > > We would have one entry point centrally for all Kafka commands.
> > > kafka <CMD> <ARGS>
> > > kafka createTopic --brokerList etc,
> > > kafka reassignPartition --brokerList etc,
> > >
> > > or execute and run the shell
> > >
> > > kafka --brokerList localhost
> > > kafka>use topicName;
> > > kafka>set acl='label';
> > >
> > > I was thinking that all calls would be initialized through
> --brokerList and
> > > the broker can tell the KafkaCommandTool what server to connect to for
> > > MetaData.
> > >
> > > Thoughts? Tomatoes?
> > >
> > > /*******************************************
> > > Joe Stein
> > > Founder, Principal Consultant
> > > Big Data Open Source Security LLC
> > > http://www.stealth.ly
> > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > > ********************************************/
> > >
> >
>
Re: Kafka Command Line & Shell
Posted by Harsha <ka...@harsha.io>.
+1 for Web Api
On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote:
> Apache Karaf has been doing this for quite a few years, albeit in Java
> not Scala. Still, their coding approach to creating a CLI probably
> captures many lessons learned over that time.
>
> Glen
>
> On 10/17/2014 08:03 PM, Joe Stein wrote:
> > Hi, I have been thinking about the ease of use for operations with Kafka.
> > We have lots of tools doing a lot of different things and they are all kind
> > of in different places.
> >
> > So, what I was thinking is to have a single interface for our tooling
> > https://issues.apache.org/jira/browse/KAFKA-1694
> >
> > This would manifest itself in two ways 1) a command line interface 2) a repl
> >
> > We would have one entry point centrally for all Kafka commands.
> > kafka <CMD> <ARGS>
> > kafka createTopic --brokerList etc,
> > kafka reassignPartition --brokerList etc,
> >
> > or execute and run the shell
> >
> > kafka --brokerList localhost
> > kafka>use topicName;
> > kafka>set acl='label';
> >
> > I was thinking that all calls would be initialized through --brokerList and
> > the broker can tell the KafkaCommandTool what server to connect to for
> > MetaData.
> >
> > Thoughts? Tomatoes?
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
> >
>
Re: Kafka Command Line & Shell
Posted by Glen Mazza <gl...@gmail.com>.
Apache Karaf has been doing this for quite a few years, albeit in Java
not Scala. Still, their coding approach to creating a CLI probably
captures many lessons learned over that time.
Glen
On 10/17/2014 08:03 PM, Joe Stein wrote:
> Hi, I have been thinking about the ease of use for operations with Kafka.
> We have lots of tools doing a lot of different things and they are all kind
> of in different places.
>
> So, what I was thinking is to have a single interface for our tooling
> https://issues.apache.org/jira/browse/KAFKA-1694
>
> This would manifest itself in two ways 1) a command line interface 2) a repl
>
> We would have one entry point centrally for all Kafka commands.
> kafka <CMD> <ARGS>
> kafka createTopic --brokerList etc,
> kafka reassignPartition --brokerList etc,
>
> or execute and run the shell
>
> kafka --brokerList localhost
> kafka>use topicName;
> kafka>set acl='label';
>
> I was thinking that all calls would be initialized through --brokerList and
> the broker can tell the KafkaCommandTool what server to connect to for
> MetaData.
>
> Thoughts? Tomatoes?
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
Re: Kafka Command Line & Shell
Posted by gs...@cloudera.com.
+1 to http interface and cli.
Http layer will make it easier to integrate with gui like Hue.
Gwen
—
Sent from Mailbox
On Fri, Oct 17, 2014 at 10:14 PM, Todd Palino <tp...@gmail.com> wrote:
> Absolutely. My suggestion of an HTTP interface was in addition to a CLI. I think the CLI should can the HTTP interface underneath to keep it simple.
> -Todd
>> On Oct 17, 2014, at 6:24 PM, Joel Koshy <jj...@gmail.com> wrote:
>>
>> +1
>> It would definitely be useful to have a CLI. We had a cursory
>> discussion on this in the past [1] but it would be useful to have a
>> full proposal describing everything the CLI should provide.
>>
>> [1] http://grokbase.com/t/kafka/dev/1435tr3pfc/command-line-tools
>>
>>> On Fri, Oct 17, 2014 at 05:12:16PM -0700, Todd Palino wrote:
>>> We've been talking about this a little internally as well. What about the idea of presenting all the admin functions through a web API interface (restful or not) complete with authentication? That would make it much easier for creating structure around Kafka without having to layer commands on top of each other.
>>>
>>> I'm not a big fan of the language specific interfaces, because they tend to complicate trying to integrate with larger systems. Consider something like AWS or Azure, where it would be much easier if there is an API interface like that.
>>>
>>> -Todd
>>>
>>>> On Oct 17, 2014, at 5:03 PM, Joe Stein <jo...@stealth.ly> wrote:
>>>>
>>>> Hi, I have been thinking about the ease of use for operations with Kafka.
>>>> We have lots of tools doing a lot of different things and they are all kind
>>>> of in different places.
>>>>
>>>> So, what I was thinking is to have a single interface for our tooling
>>>> https://issues.apache.org/jira/browse/KAFKA-1694
>>>>
>>>> This would manifest itself in two ways 1) a command line interface 2) a repl
>>>>
>>>> We would have one entry point centrally for all Kafka commands.
>>>> kafka <CMD> <ARGS>
>>>> kafka createTopic --brokerList etc,
>>>> kafka reassignPartition --brokerList etc,
>>>>
>>>> or execute and run the shell
>>>>
>>>> kafka --brokerList localhost
>>>> kafka>use topicName;
>>>> kafka>set acl='label';
>>>>
>>>> I was thinking that all calls would be initialized through --brokerList and
>>>> the broker can tell the KafkaCommandTool what server to connect to for
>>>> MetaData.
>>>>
>>>> Thoughts? Tomatoes?
>>>>
>>>> /*******************************************
>>>> Joe Stein
>>>> Founder, Principal Consultant
>>>> Big Data Open Source Security LLC
>>>> http://www.stealth.ly
>>>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>>>> ********************************************/
>>
Re: Kafka Command Line & Shell
Posted by Todd Palino <tp...@gmail.com>.
Absolutely. My suggestion of an HTTP interface was in addition to a CLI. I think the CLI should can the HTTP interface underneath to keep it simple.
-Todd
> On Oct 17, 2014, at 6:24 PM, Joel Koshy <jj...@gmail.com> wrote:
>
> +1
> It would definitely be useful to have a CLI. We had a cursory
> discussion on this in the past [1] but it would be useful to have a
> full proposal describing everything the CLI should provide.
>
> [1] http://grokbase.com/t/kafka/dev/1435tr3pfc/command-line-tools
>
>> On Fri, Oct 17, 2014 at 05:12:16PM -0700, Todd Palino wrote:
>> We've been talking about this a little internally as well. What about the idea of presenting all the admin functions through a web API interface (restful or not) complete with authentication? That would make it much easier for creating structure around Kafka without having to layer commands on top of each other.
>>
>> I'm not a big fan of the language specific interfaces, because they tend to complicate trying to integrate with larger systems. Consider something like AWS or Azure, where it would be much easier if there is an API interface like that.
>>
>> -Todd
>>
>>> On Oct 17, 2014, at 5:03 PM, Joe Stein <jo...@stealth.ly> wrote:
>>>
>>> Hi, I have been thinking about the ease of use for operations with Kafka.
>>> We have lots of tools doing a lot of different things and they are all kind
>>> of in different places.
>>>
>>> So, what I was thinking is to have a single interface for our tooling
>>> https://issues.apache.org/jira/browse/KAFKA-1694
>>>
>>> This would manifest itself in two ways 1) a command line interface 2) a repl
>>>
>>> We would have one entry point centrally for all Kafka commands.
>>> kafka <CMD> <ARGS>
>>> kafka createTopic --brokerList etc,
>>> kafka reassignPartition --brokerList etc,
>>>
>>> or execute and run the shell
>>>
>>> kafka --brokerList localhost
>>> kafka>use topicName;
>>> kafka>set acl='label';
>>>
>>> I was thinking that all calls would be initialized through --brokerList and
>>> the broker can tell the KafkaCommandTool what server to connect to for
>>> MetaData.
>>>
>>> Thoughts? Tomatoes?
>>>
>>> /*******************************************
>>> Joe Stein
>>> Founder, Principal Consultant
>>> Big Data Open Source Security LLC
>>> http://www.stealth.ly
>>> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
>>> ********************************************/
>
Re: Kafka Command Line & Shell
Posted by Joel Koshy <jj...@gmail.com>.
+1
It would definitely be useful to have a CLI. We had a cursory
discussion on this in the past [1] but it would be useful to have a
full proposal describing everything the CLI should provide.
[1] http://grokbase.com/t/kafka/dev/1435tr3pfc/command-line-tools
On Fri, Oct 17, 2014 at 05:12:16PM -0700, Todd Palino wrote:
> We've been talking about this a little internally as well. What about the idea of presenting all the admin functions through a web API interface (restful or not) complete with authentication? That would make it much easier for creating structure around Kafka without having to layer commands on top of each other.
>
> I'm not a big fan of the language specific interfaces, because they tend to complicate trying to integrate with larger systems. Consider something like AWS or Azure, where it would be much easier if there is an API interface like that.
>
> -Todd
>
> > On Oct 17, 2014, at 5:03 PM, Joe Stein <jo...@stealth.ly> wrote:
> >
> > Hi, I have been thinking about the ease of use for operations with Kafka.
> > We have lots of tools doing a lot of different things and they are all kind
> > of in different places.
> >
> > So, what I was thinking is to have a single interface for our tooling
> > https://issues.apache.org/jira/browse/KAFKA-1694
> >
> > This would manifest itself in two ways 1) a command line interface 2) a repl
> >
> > We would have one entry point centrally for all Kafka commands.
> > kafka <CMD> <ARGS>
> > kafka createTopic --brokerList etc,
> > kafka reassignPartition --brokerList etc,
> >
> > or execute and run the shell
> >
> > kafka --brokerList localhost
> > kafka>use topicName;
> > kafka>set acl='label';
> >
> > I was thinking that all calls would be initialized through --brokerList and
> > the broker can tell the KafkaCommandTool what server to connect to for
> > MetaData.
> >
> > Thoughts? Tomatoes?
> >
> > /*******************************************
> > Joe Stein
> > Founder, Principal Consultant
> > Big Data Open Source Security LLC
> > http://www.stealth.ly
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > ********************************************/
Re: Kafka Command Line & Shell
Posted by Todd Palino <tp...@gmail.com>.
We've been talking about this a little internally as well. What about the idea of presenting all the admin functions through a web API interface (restful or not) complete with authentication? That would make it much easier for creating structure around Kafka without having to layer commands on top of each other.
I'm not a big fan of the language specific interfaces, because they tend to complicate trying to integrate with larger systems. Consider something like AWS or Azure, where it would be much easier if there is an API interface like that.
-Todd
> On Oct 17, 2014, at 5:03 PM, Joe Stein <jo...@stealth.ly> wrote:
>
> Hi, I have been thinking about the ease of use for operations with Kafka.
> We have lots of tools doing a lot of different things and they are all kind
> of in different places.
>
> So, what I was thinking is to have a single interface for our tooling
> https://issues.apache.org/jira/browse/KAFKA-1694
>
> This would manifest itself in two ways 1) a command line interface 2) a repl
>
> We would have one entry point centrally for all Kafka commands.
> kafka <CMD> <ARGS>
> kafka createTopic --brokerList etc,
> kafka reassignPartition --brokerList etc,
>
> or execute and run the shell
>
> kafka --brokerList localhost
> kafka>use topicName;
> kafka>set acl='label';
>
> I was thinking that all calls would be initialized through --brokerList and
> the broker can tell the KafkaCommandTool what server to connect to for
> MetaData.
>
> Thoughts? Tomatoes?
>
> /*******************************************
> Joe Stein
> Founder, Principal Consultant
> Big Data Open Source Security LLC
> http://www.stealth.ly
> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/