You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Su She <su...@gmail.com> on 2015/01/20 02:46:42 UTC
java.nio.channels.ClosedChannelException...Firewall Issue?
Hello Everyone,
Thank you for the help!
Preface: I've created producers/consumers before and they have worked. I
have also made consumers/producers using java programs, but they have all
been locally.
1) I have a Zookeeper/Kafka Server running on an EC2 instance called "A"
2) I started the Zookeeper/Kafka Server on A and created a topic "test"
like it says on the kafka documentation
3) I then started a console consumer on A like the documentation states for
test.
4) I then downloaded Kafka on EC2 instance called "B" and created a console
producer with broker with broker-list public.ip.of.A:9092 and topic test.
5) Once I start publishing messages this is the message I get (what's worse
is that I can't cntrl C to stop and have to exit putty):
WARN Fetching topic metadata with correlation id 16 for topics [Set(test)]
from broker [id:0,host:54.183.40.224,port:9092] failed
(kafka.client.ClientUtils$)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
at
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:73)
at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
at
kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
at
kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at
kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)
at kafka.utils.Utils$.swallow(Utils.scala:172)
at kafka.utils.Logging$class.swallowError(Logging.scala:106)
at kafka.utils.Utils$.swallowError(Utils.scala:45)
at
kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
at
kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
at
kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:93)
at
kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
[2015-01-20 01:29:28,289] ERROR fetching topic metadata for topics
[Set(test)] from broker [ArrayBuffer(id:0,host:
Meanwhile in ZK...not sure if this happened as soon as I started publishing
messages:
Error:KeeperErrorCode = NoNode for /consumers/console-consumer-2615/offsets
(org.apache.zookeeper.server.PrepRequestProcessor)
Any tips/suggestions are greatly appreciated!
Re: java.nio.channels.ClosedChannelException...Firewall Issue?
Posted by Su She <su...@gmail.com>.
Hi Jaikiran,
Thanks for the reply!
1) I started Kafka server on instance A by simply downloading
Kafka_2.10-0.8.2-beta.tgz from the kafka website, and using the scripts
mentioned here: http://kafka.apache.org/documentation.html#introduction.
This is the same way I downloaded Kafka on B, except I only started the
producer, not the ZK or Server.
2) How do I check to see if the services are bound to non localhost IP?
3) I believe it is firewall because I started 2 instances (C and D) using
the AWS Quick Start (
https://s3.amazonaws.com/quickstart-reference/cloudera/hadoop/latest/doc/Cloudera_EDH_on_AWS.pdf).
I was able to download Kafka on C and D and pass messages using Kafka
Producer/Consumer from D to C using the private/public ip address of C
(which was the instance I was running the consumer/ZK/Server on). Since I
used AWS Quick Start both of these were in a private subnet.
Edit: Woops, I saw I said B was an ec2-instance, B is a gateway, and I
downloaded Kafka same way I downloaded it on EC2 and I started a producer
on the gateway hoping to send it to my EC2 instance. I can ping A (in a
private subnet) from my gateway fine. I have pretty much changed all
security settings on A so that it essentially has none, but still in a
private subnet ( so can't ssh into it still from putty, can only ssh via
the cluster launcher). I can't lessen the security anymore on instance A.
Hope that wasn't too confusing, really appreciate the help. Thank you!
Best,
Su
On Mon, Jan 19, 2015 at 7:58 PM, Jaikiran Pai <ja...@gmail.com>
wrote:
> Hi Su,
>
> How exactly did you start the Kafka server on instance "A"? Are you sure
> the services on it are bound to non localhost IP? What does the following
> command result from instance B:
>
> telnet public.ip.of.A 9092
>
>
> -Jaikiran
>
> On Tuesday 20 January 2015 07:16 AM, Su She wrote:
>
>> Hello Everyone,
>>
>> Thank you for the help!
>>
>> Preface: I've created producers/consumers before and they have worked. I
>> have also made consumers/producers using java programs, but they have all
>> been locally.
>>
>> 1) I have a Zookeeper/Kafka Server running on an EC2 instance called "A"
>>
>> 2) I started the Zookeeper/Kafka Server on A and created a topic "test"
>> like it says on the kafka documentation
>>
>> 3) I then started a console consumer on A like the documentation states
>> for
>> test.
>>
>> 4) I then downloaded Kafka on EC2 instance called "B" and created a
>> console
>> producer with broker with broker-list public.ip.of.A:9092 and topic test.
>>
>> 5) Once I start publishing messages this is the message I get (what's
>> worse
>> is that I can't cntrl C to stop and have to exit putty):
>>
>> WARN Fetching topic metadata with correlation id 16 for topics [Set(test)]
>> from broker [id:0,host:54.183.40.224,port:9092] failed
>> (kafka.client.ClientUtils$)
>>
>> java.nio.channels.ClosedChannelException
>>
>> at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
>>
>> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.
>> scala:74)
>>
>> at
>> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$
>> doSend(SyncProducer.scala:73)
>>
>> at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
>>
>> at
>> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
>>
>> at
>> kafka.producer.BrokerPartitionInfo.updateInfo(
>> BrokerPartitionInfo.scala:82)
>>
>> at
>> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(
>> DefaultEventHandler.scala:78)
>>
>> at kafka.utils.Utils$.swallow(Utils.scala:172)
>>
>> at kafka.utils.Logging$class.swallowError(Logging.scala:106)
>>
>> at kafka.utils.Utils$.swallowError(Utils.scala:45)
>>
>> at
>> kafka.producer.async.DefaultEventHandler.handle(
>> DefaultEventHandler.scala:78)
>>
>> at
>> kafka.producer.async.ProducerSendThread.tryToHandle(
>> ProducerSendThread.scala:104)
>>
>> at
>> kafka.producer.async.ProducerSendThread.processEvents(
>> ProducerSendThread.scala:93)
>>
>> at
>> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>>
>> [2015-01-20 01:29:28,289] ERROR fetching topic metadata for topics
>> [Set(test)] from broker [ArrayBuffer(id:0,host:
>>
>> Meanwhile in ZK...not sure if this happened as soon as I started
>> publishing
>> messages:
>>
>> Error:KeeperErrorCode = NoNode for /consumers/console-consumer-
>> 2615/offsets
>> (org.apache.zookeeper.server.PrepRequestProcessor)
>>
>> Any tips/suggestions are greatly appreciated!
>>
>>
>
Re: java.nio.channels.ClosedChannelException...Firewall Issue?
Posted by Jaikiran Pai <ja...@gmail.com>.
Hi Su,
How exactly did you start the Kafka server on instance "A"? Are you sure
the services on it are bound to non localhost IP? What does the
following command result from instance B:
telnet public.ip.of.A 9092
-Jaikiran
On Tuesday 20 January 2015 07:16 AM, Su She wrote:
> Hello Everyone,
>
> Thank you for the help!
>
> Preface: I've created producers/consumers before and they have worked. I
> have also made consumers/producers using java programs, but they have all
> been locally.
>
> 1) I have a Zookeeper/Kafka Server running on an EC2 instance called "A"
>
> 2) I started the Zookeeper/Kafka Server on A and created a topic "test"
> like it says on the kafka documentation
>
> 3) I then started a console consumer on A like the documentation states for
> test.
>
> 4) I then downloaded Kafka on EC2 instance called "B" and created a console
> producer with broker with broker-list public.ip.of.A:9092 and topic test.
>
> 5) Once I start publishing messages this is the message I get (what's worse
> is that I can't cntrl C to stop and have to exit putty):
>
> WARN Fetching topic metadata with correlation id 16 for topics [Set(test)]
> from broker [id:0,host:54.183.40.224,port:9092] failed
> (kafka.client.ClientUtils$)
>
> java.nio.channels.ClosedChannelException
>
> at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
>
> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:74)
>
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:73)
>
> at kafka.producer.SyncProducer.send(SyncProducer.scala:114)
>
> at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58)
>
> at
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
>
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp(DefaultEventHandler.scala:78)
>
> at kafka.utils.Utils$.swallow(Utils.scala:172)
>
> at kafka.utils.Logging$class.swallowError(Logging.scala:106)
>
> at kafka.utils.Utils$.swallowError(Utils.scala:45)
>
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:78)
>
> at
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104)
>
> at
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:93)
>
> at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44)
>
> [2015-01-20 01:29:28,289] ERROR fetching topic metadata for topics
> [Set(test)] from broker [ArrayBuffer(id:0,host:
>
> Meanwhile in ZK...not sure if this happened as soon as I started publishing
> messages:
>
> Error:KeeperErrorCode = NoNode for /consumers/console-consumer-2615/offsets
> (org.apache.zookeeper.server.PrepRequestProcessor)
>
> Any tips/suggestions are greatly appreciated!
>