You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Jeff Gong <j....@gmail.com> on 2015/07/07 19:43:11 UTC
Launch Kafka/Zookeeper servers without using command line
Hi all,
Is it possible to launch Kafka/Zookeeper servers via some part of the API?
I noticed in this:
http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/
that it seems to be possible to create a ZK instance and add a topic to it,
but is this the same as launching the ZK server as we would in the command
line?
Similarly, would it be possible to do this for a Kafka server and have it
connect to an existing ZK connection?
Thanks!
Re: Launch Kafka/Zookeeper servers without using command line
Posted by Rakesh Vidyadharan <rv...@gracenote.com>.
Hello,
I tried using the sources for unit tests, but have been unsuccessful in
getting it to work with the low level API (kafka version 0.8.2.2). Our
code is modelled on the Simple Consumer API code, and the error always
occurs in the findLeader code.
SEVERE: Error communicating with broker:localhost:53412 attempting to find
a leader for channel:catapult.schedule.day, partition: 0.Reason:
java.nio.channels.ClosedChannelException
Looking at the logs the embedded server(s) generate, I see these types of
messages which seemed interesting for this particular issue (first one is
that first org.apache.zookeeper.ClientCnxn always seems to fail on the
currently assigned port, and later on a bunch of KeeperErrorCode = NoNode
for /brokers etc):
[main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection,
connectString=localhost:53411 sessionTimeout=6000
watcher=org.I0Itec.zkclient.ZkClient@14be60b
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Opening socket connection to server localhost/0:0:0:0:0:0:0:1:53411. Will
not attempt to authenticate using SASL (unknown error)
[main-SendThread(localhost:53411)] WARN org.apache.zookeeper.ClientCnxn -
Session 0x0 for server null, unexpected error, closing socket connection
and attempting reconnect
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.ja
va:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Opening socket connection to server localhost/127.0.0.1:53411. Will not
attempt to authenticate using SASL (unknown error)
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Socket connection established to localhost/127.0.0.1:53411, initiating
session
[NIOServerCxn.Factory:localhost/127.0.0.1:53411] INFO
org.apache.zookeeper.server.NIOServerCnxnFactory - Accepted socket
connection from /127.0.0.1:53414
[NIOServerCxn.Factory:localhost/127.0.0.1:53411] INFO
org.apache.zookeeper.server.ZooKeeperServer - Client attempting to
establish new session at /127.0.0.1:53414
[SyncThread:0] INFO org.apache.zookeeper.server.persistence.FileTxnLog -
Creating new log file: log.1
[SyncThread:0] INFO org.apache.zookeeper.server.ZooKeeperServer -
Established session 0x15006291f440000 with negotiated timeout 6000 for
client /127.0.0.1:53414
[main-SendThread(localhost:53411)] INFO org.apache.zookeeper.ClientCnxn -
Session establishment complete on server localhost/127.0.0.1:53411,
sessionid = 0x15006291f440000, negotiated timeout = 6000
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:create
cxid:0x4 zxid:0x3 txntype:-1 reqpath:n/a Error Path:/brokers
Error:KeeperErrorCode = NoNode for /brokers
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:create
cxid:0xa zxid:0x7 txntype:-1 reqpath:n/a Error Path:/config
Error:KeeperErrorCode = NoNode for /config
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:create
cxid:0x10 zxid:0xb txntype:-1 reqpath:n/a Error Path:/admin
Error:KeeperErrorCode = NoNode for /admin
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:setData
cxid:0x1a zxid:0xf txntype:-1 reqpath:n/a Error Path:/controller_epoch
Error:KeeperErrorCode = NoNode for /controller_epoch
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Got user-level
KeeperException when processing sessionid:0x15006291f440000 type:delete
cxid:0x27 zxid:0x11 txntype:-1 reqpath:n/a Error
Path:/admin/preferred_replica_election Error:KeeperErrorCode = NoNode for
/admin/preferred_replica_election
[ProcessThread(sid:0 cport:-1):] INFO
org.apache.zookeeper.server.PrepRequestProcessor - Processed session
termination for sessionid: 0x15006291f440000
Thanks
Rakesh
On 07/07/2015 21:16, "Gwen Shapira" <gs...@cloudera.com> wrote:
>Is this for unit tests? Or do you need an embedded Kafka / ZK inside an
>app?
>Or do you mean launching an external Kafka cluster, but without command
>line?
>
>For unit tests / embedded, you can see this example:
>https://gist.github.com/vmarcinko/e4e58910bcb77dac16e9
>
>To start an actual Kafka server through an API call, you typically
>need some kind of agent to take the API call and launch Kafka.
>Cloudera Manager has an API that can do this, and I'm sure Ambari can
>do the same.
>
>Gwen
>
>On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j....@gmail.com> wrote:
>> Hi all,
>>
>> Is it possible to launch Kafka/Zookeeper servers via some part of the
>>API?
>> I noticed in this:
>>
>>http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-
>>kafka-from-the-ide-using-api/
>> that it seems to be possible to create a ZK instance and add a topic to
>>it,
>> but is this the same as launching the ZK server as we would in the
>>command
>> line?
>>
>> Similarly, would it be possible to do this for a Kafka server and have
>>it
>> connect to an existing ZK connection?
>>
>> Thanks!
Re: Launch Kafka/Zookeeper servers without using command line
Posted by Gwen Shapira <gs...@cloudera.com>.
Is this for unit tests? Or do you need an embedded Kafka / ZK inside an app?
Or do you mean launching an external Kafka cluster, but without command line?
For unit tests / embedded, you can see this example:
https://gist.github.com/vmarcinko/e4e58910bcb77dac16e9
To start an actual Kafka server through an API call, you typically
need some kind of agent to take the API call and launch Kafka.
Cloudera Manager has an API that can do this, and I'm sure Ambari can
do the same.
Gwen
On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j....@gmail.com> wrote:
> Hi all,
>
> Is it possible to launch Kafka/Zookeeper servers via some part of the API?
> I noticed in this:
> http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/
> that it seems to be possible to create a ZK instance and add a topic to it,
> but is this the same as launching the ZK server as we would in the command
> line?
>
> Similarly, would it be possible to do this for a Kafka server and have it
> connect to an existing ZK connection?
>
> Thanks!
Re: Launch Kafka/Zookeeper servers without using command line
Posted by Rad Gruchalski <ra...@gruchalski.com>.
Hi everyone,
Same errors can be seen when using embedded kafka and embedded zookeeper in unit tests. They’re absolutely normal. As long as you see a successful connection, it’s all good!
Kind regards,
Radek Gruchalski
radek@gruchalski.com (mailto:radek@gruchalski.com)
(mailto:radek@gruchalski.com)
de.linkedin.com/in/radgruchalski/ (http://de.linkedin.com/in/radgruchalski/)
Confidentiality:
This communication is intended for the above-named person and may be confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must you copy or show it to anyone; please delete/destroy and inform the sender immediately.
On Friday, 10 July 2015 at 19:09, Jeff Gong wrote:
> To follow up and provide a little more context on my second bullet point,
> when I run any command for the first time on command line that requires
> connecting to this code instantiated ZK server I get the specific error:
>
> > bin/kafka-topics.sh (http://kafka-topics.sh) --list --zookeeper localhost:2181
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>
>
> But after that it looks like a reconnect is attempted, which succeeds and
> I'll get a list of my topics or my console consumer will succeed, etc. If I
> were to run any of those commands a second time, this error then
> disappears. Anyone have any ideas?
>
> On Thu, Jul 9, 2015 at 4:20 PM, Jeff Gong <j.gong16@gmail.com (mailto:j.gong16@gmail.com)> wrote:
>
> > Hey Gwen,
> >
> > Thanks for your response. The example was very helpful and exactly what I
> > was looking for! I went ahead and used that as an example and while most
> > things are working as I wanted, there are a few issues I'd like to follow
> > up on:
> >
> > 1. If I launch a Zookeeper/Kafka server from code instead of command line,
> > most of the helpful debugging/status messages will not be displayed -- is
> > there any way I can retain this functionality?
> >
> > 2. I created a scenario where I successfully launched a Zookeeper instance
> > from code, then connected to that Zookeeper server using a Kafka Server via
> > command line. I then instantiated both a Kafka-Console-Consumer and
> > Kafka-Console-Producer with the topic 'test'. I was able to send messages
> > to the consumer as expected, but I noticed that when I try to connect the
> > console consumer to the server I always get a 'Connection Refused' error to
> > my Kafka Broker. Any ideas why this might be occuring?
> >
> > Best,
> > Jeff
> >
> >
> >
> > On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j.gong16@gmail.com (mailto:j.gong16@gmail.com)> wrote:
> >
> > > Hi all,
> > >
> > > Is it possible to launch Kafka/Zookeeper servers via some part of the
> > > API? I noticed in this:
> > > http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/
> > > that it seems to be possible to create a ZK instance and add a topic to it,
> > > but is this the same as launching the ZK server as we would in the command
> > > line?
> > >
> > > Similarly, would it be possible to do this for a Kafka server and have it
> > > connect to an existing ZK connection?
> > >
> > > Thanks!
Re: Launch Kafka/Zookeeper servers without using command line
Posted by Mayuresh Gharat <gh...@gmail.com>.
Hi Jeff,
I haven't tried this out, but I am planning to. Just a quick question : We
have TestHarness in Kafka that brings up Kafka and Zookeeper and also tears
them down. Have you tried using it?
Thanks,
Mayuresh
On Fri, Jul 10, 2015 at 10:09 AM, Jeff Gong <j....@gmail.com> wrote:
> To follow up and provide a little more context on my second bullet point,
> when I run any command for the first time on command line that requires
> connecting to this code instantiated ZK server I get the specific error:
>
> > bin/kafka-topics.sh --list --zookeeper localhost:2181
>
> java.net.ConnectException: Connection refused
>
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>
> at
>
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
>
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>
>
> But after that it looks like a reconnect is attempted, which succeeds and
> I'll get a list of my topics or my console consumer will succeed, etc. If I
> were to run any of those commands a second time, this error then
> disappears. Anyone have any ideas?
>
> On Thu, Jul 9, 2015 at 4:20 PM, Jeff Gong <j....@gmail.com> wrote:
>
> > Hey Gwen,
> >
> > Thanks for your response. The example was very helpful and exactly what I
> > was looking for! I went ahead and used that as an example and while most
> > things are working as I wanted, there are a few issues I'd like to follow
> > up on:
> >
> > 1. If I launch a Zookeeper/Kafka server from code instead of command
> line,
> > most of the helpful debugging/status messages will not be displayed -- is
> > there any way I can retain this functionality?
> >
> > 2. I created a scenario where I successfully launched a Zookeeper
> instance
> > from code, then connected to that Zookeeper server using a Kafka Server
> via
> > command line. I then instantiated both a Kafka-Console-Consumer and
> > Kafka-Console-Producer with the topic 'test'. I was able to send messages
> > to the consumer as expected, but I noticed that when I try to connect the
> > console consumer to the server I always get a 'Connection Refused' error
> to
> > my Kafka Broker. Any ideas why this might be occuring?
> >
> > Best,
> > Jeff
> >
> >
> >
> > On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j....@gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> Is it possible to launch Kafka/Zookeeper servers via some part of the
> >> API? I noticed in this:
> >>
> http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/
> >> that it seems to be possible to create a ZK instance and add a topic to
> it,
> >> but is this the same as launching the ZK server as we would in the
> command
> >> line?
> >>
> >> Similarly, would it be possible to do this for a Kafka server and have
> it
> >> connect to an existing ZK connection?
> >>
> >> Thanks!
> >>
> >
> >
>
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
Re: Launch Kafka/Zookeeper servers without using command line
Posted by Jeff Gong <j....@gmail.com>.
To follow up and provide a little more context on my second bullet point,
when I run any command for the first time on command line that requires
connecting to this code instantiated ZK server I get the specific error:
> bin/kafka-topics.sh --list --zookeeper localhost:2181
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
But after that it looks like a reconnect is attempted, which succeeds and
I'll get a list of my topics or my console consumer will succeed, etc. If I
were to run any of those commands a second time, this error then
disappears. Anyone have any ideas?
On Thu, Jul 9, 2015 at 4:20 PM, Jeff Gong <j....@gmail.com> wrote:
> Hey Gwen,
>
> Thanks for your response. The example was very helpful and exactly what I
> was looking for! I went ahead and used that as an example and while most
> things are working as I wanted, there are a few issues I'd like to follow
> up on:
>
> 1. If I launch a Zookeeper/Kafka server from code instead of command line,
> most of the helpful debugging/status messages will not be displayed -- is
> there any way I can retain this functionality?
>
> 2. I created a scenario where I successfully launched a Zookeeper instance
> from code, then connected to that Zookeeper server using a Kafka Server via
> command line. I then instantiated both a Kafka-Console-Consumer and
> Kafka-Console-Producer with the topic 'test'. I was able to send messages
> to the consumer as expected, but I noticed that when I try to connect the
> console consumer to the server I always get a 'Connection Refused' error to
> my Kafka Broker. Any ideas why this might be occuring?
>
> Best,
> Jeff
>
>
>
> On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j....@gmail.com> wrote:
>
>> Hi all,
>>
>> Is it possible to launch Kafka/Zookeeper servers via some part of the
>> API? I noticed in this:
>> http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/
>> that it seems to be possible to create a ZK instance and add a topic to it,
>> but is this the same as launching the ZK server as we would in the command
>> line?
>>
>> Similarly, would it be possible to do this for a Kafka server and have it
>> connect to an existing ZK connection?
>>
>> Thanks!
>>
>
>
Re: Launch Kafka/Zookeeper servers without using command line
Posted by Jeff Gong <j....@gmail.com>.
Hey Gwen,
Thanks for your response. The example was very helpful and exactly what I
was looking for! I went ahead and used that as an example and while most
things are working as I wanted, there are a few issues I'd like to follow
up on:
1. If I launch a Zookeeper/Kafka server from code instead of command line,
most of the helpful debugging/status messages will not be displayed -- is
there any way I can retain this functionality?
2. I created a scenario where I successfully launched a Zookeeper instance
from code, then connected to that Zookeeper server using a Kafka Server via
command line. I then instantiated both a Kafka-Console-Consumer and
Kafka-Console-Producer with the topic 'test'. I was able to send messages
to the consumer as expected, but I noticed that when I try to connect the
console consumer to the server I always get a 'Connection Refused' error to
my Kafka Broker. Any ideas why this might be occuring?
Best,
Jeff
On Tue, Jul 7, 2015 at 10:43 AM, Jeff Gong <j....@gmail.com> wrote:
> Hi all,
>
> Is it possible to launch Kafka/Zookeeper servers via some part of the API?
> I noticed in this:
> http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/
> that it seems to be possible to create a ZK instance and add a topic to it,
> but is this the same as launching the ZK server as we would in the command
> line?
>
> Similarly, would it be possible to do this for a Kafka server and have it
> connect to an existing ZK connection?
>
> Thanks!
>