You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by 傅駿浩 <ja...@yahoo.com.tw> on 2014/06/12 13:04:30 UTC
[Issue] Cannot connect to Kafka cluster on GCE from local machine
Hi all,
I want to use the KafkaSpout(where is https://github.com/apache/incubator-storm/tree/master/external/storm-kafka) to read data from kafka. But my kafka and zookeeper cluster are builted on GCE(google cloud engine) and my storm testing is on local machine. The problem is I cannot set public IP by kafka config - server.properties to reach from my local(out of kafka's domain) storm machines.
***My server.properties is as follows:
host.name=[Brokers's Private IP]
advertise.port=9092
advertise.host.name=[Brokers's Public IP]
***My KafkaSpout's config in Java is as follows:
BrokerHosts a = new ZkHosts("[IP1],[IP2],[IP3]","/brokers");
SpoutConfig kafkaConf = new SpoutConfig(a, topic, "/stormkafka", topic);//Using [topic] as id
kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
kafkaConf.zkServers = new ArrayList<String>() {
{
add("[IP1]");
add("[IP1]");
add("[IP1]");
}
};
kafkaConf.zkPort = 2181;
kafkaConf.forceFromStart = true;
ps. I have 3-node kafka cluster and 3-node zookeeper cluster on 3 machines. That's to say, Each machine with a broker and zookeeper node
======================================
So the problem is it seens not to connect from the domain different from kafka's domain, e.g.my local storm machine. This is a big problem if I need to communcate with kafka out of its domain. The error exception is as follows, and can anyone help me with this issue. How can I config it in a right way so that I can subscribe topic of kafka from my local machine(storm).
Thanks a lot :)
java.lang.RuntimeException: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:83) ~[storm-kafka-0.8-plus-0.4.0.jar:na]
at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45) ~[storm-kafka-0.8-plus-0.4.0.jar:na]
at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:118) ~[storm-kafka-0.8-plus-0.4.0.jar:na]
at backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116$fn__5145.invoke(executor.clj:562) ~[na:na]
at backtype.storm.util$async_loop$fn__390.invoke(util.clj:433) ~[na:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
Caused by: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_60]
at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_60]
at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_60]
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670) ~[na:1.7.0_60]
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:124) ~[kafka_2.10-0.8.1.1.jar:na]
at kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79) ~[kafka_2.10-0.8.1.1.jar:na]
at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:55) ~[storm-kafka-0.8-plus-0.4.0.jar:na]
at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:45) ~[storm-kafka-0.8-plus-0.4.0.jar:na]
at storm.kafka.PartitionManager.<init>(PartitionManager.java:77) ~[storm-kafka-0.8-plus-0.4.0.jar:na]
at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78) ~[storm-kafka-0.8-plus-0.4.0.jar:na]
... 6 common frames omitted
Best regards,
James Fu
Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine
Posted by Roger Hoover <ro...@gmail.com>.
Actually, there wasn't a way to do it prior to 0.8.1
On Fri, Jun 13, 2014 at 3:30 PM, Roger Hoover <ro...@gmail.com>
wrote:
> Yes, I believe that prior to Kafka 0.8 there was no easy way for external
> clients to talk to Kafka brokers running in a cloud environment.
>
> I wrote a blog post about it here:
>
>
> https://medium.com/hadoop-2/running-kafka-in-a-hybrid-cloud-environment-17a8f3cfc284
>
>
> On Fri, Jun 13, 2014 at 10:08 AM, <ja...@yahoo.com.tw> wrote:
>
>> Hi,
>> Thank you very much. Your reply make me clear with Kafka and cloud
>> environment a lot. And it makes me remind that it seems
>> advertised.host.name is available after Kafka 0.8. Perhaps we can't
>> access Kafka on GCE from local machine before, is it correct?
>>
>> Best regards,
>> James
>>
>>
>>
>> > Roger Hoover <ro...@gmail.com> 於 2014/6/14 上午12:32 寫道：
>> >
>> > I wouldn't say that Kafka's making it difficult. The cloud environment
>> is
>> > making it difficult. The VM that the Kafka broker is running on can
>> only
>> > see it's private IP (at the OS level) so you have to add the
>> > advertised.host.name config so that it knows what public IP is
>> assigned to
>> > it.
>> >
>> >
>> >> On Fri, Jun 13, 2014 at 8:32 AM, <ja...@yahoo.com.tw> wrote:
>> >>
>> >> Hi,
>> >> I found this caused by broker's config. I miss typed
>> advertise.host.name.
>> >> Actually it should be advertised.host.name.
>> >>
>> >> However, I still don't know what's different between host.name and
>> >> advertised.host.name. Why Kafka makes it so difficult to connect from
>> >> public ip. Thanks your reply.
>> >>
>> >> Best regards,
>> >> James
>> >>
>> >>
>> >>
>> >>
>> >>> Jun Rao <ju...@gmail.com> 於 2014/6/13 下午10:22 寫道：
>> >>>
>> >>> Could you telnet to the public ip from your local machine?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Jun
>> >>>
>> >>>
>> >>>> On Thu, Jun 12, 2014 at 4:04 AM, 傅駿浩 <ja...@yahoo.com.tw> wrote:
>> >>>>
>> >>>> Hi all,
>> >>>>
>> >>>> I want to use the KafkaSpout(where is
>> >>
>> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka
>> >> )
>> >>>> to read data from kafka. But my kafka and zookeeper cluster are
>> builted
>> >> on
>> >>>> GCE(google cloud engine) and my storm testing is on local machine.
>> The
>> >>>> problem is I cannot set public IP by kafka config -
>> server.properties to
>> >>>> reach from my local(out of kafka's domain) storm machines.
>> >>>> ***My server.properties is as follows:
>> >>>> host.name=[Brokers's Private IP]
>> >>>> advertise.port=9092
>> >>>> advertise.host.name=[Brokers's Public IP]
>> >>>>
>> >>>> ***My KafkaSpout's config in Java is as follows:
>> >>>> BrokerHosts a = new ZkHosts("[IP1],[IP2],[IP3]","/brokers");
>> >>>> SpoutConfig kafkaConf = new SpoutConfig(a, topic,
>> "/stormkafka",
>> >>>> topic);//Using [topic] as id
>> >>>> kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>> >>>> kafkaConf.zkServers = new ArrayList<String>() {
>> >>>> {
>> >>>> add("[IP1]");
>> >>>> add("[IP1]");
>> >>>> add("[IP1]");
>> >>>> }
>> >>>> };
>> >>>> kafkaConf.zkPort = 2181;
>> >>>> kafkaConf.forceFromStart = true;
>> >>>> ps. I have 3-node kafka cluster and 3-node zookeeper cluster on 3
>> >>>> machines. That's to say, Each machine with a broker and zookeeper
>> node
>> >>>>
>> >>>> ======================================
>> >>>> So the problem is it seens not to connect from the domain different
>> from
>> >>>> kafka's domain, e.g.my local storm machine. This is a big problem
>> if I
>> >>>> need to communcate with kafka out of its domain. The error exception
>> is
>> >> as
>> >>>> follows, and can anyone help me with this issue. How can I config it
>> in
>> >> a
>> >>>> right way so that I can subscribe topic of kafka from my local
>> >>>> machine(storm).
>> >>>> Thanks a lot :)
>> >>>>
>> >>>>
>> >>>>
>> >>>> java.lang.RuntimeException: java.net.NoRouteToHostException:
>> >> 找不到通往該主機的路由路徑
>> >>>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:83)
>> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> >>>> at
>> >> storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45)
>> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> >>>> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:118)
>> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> >>>> at
>> >>
>> backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116$fn__5145.invoke(executor.clj:562)
>> >>>> ~[na:na]
>> >>>> at backtype.storm.util$async_loop$fn__390.invoke(util.clj:433)
>> ~[na:na]
>> >>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>> >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
>> >>>> Caused by: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
>> >>>> at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_60]
>> >>>> at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_60]
>> >>>> at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_60]
>> >>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
>> >>>> ~[na:1.7.0_60]
>> >>>> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
>> >>>> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
>> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
>> >>>> at
>> >>
>> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142)
>> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
>> >>>> at
>> >>
>> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
>> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
>> >>>> at
>> >>>>
>> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:124)
>> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
>> >>>> at
>> >>
>> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
>> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
>> >>>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:55)
>> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> >>>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:45)
>> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> >>>> at storm.kafka.PartitionManager.<init>(PartitionManager.java:77)
>> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> >>>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78)
>> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> >>>> ... 6 common frames omitted
>> >>>>
>> >>>> Best regards,
>> >>>> James Fu
>> >>
>>
>
>
Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine
Posted by Roger Hoover <ro...@gmail.com>.
Yes, I believe that prior to Kafka 0.8 there was no easy way for external
clients to talk to Kafka brokers running in a cloud environment.
I wrote a blog post about it here:
https://medium.com/hadoop-2/running-kafka-in-a-hybrid-cloud-environment-17a8f3cfc284
On Fri, Jun 13, 2014 at 10:08 AM, <ja...@yahoo.com.tw> wrote:
> Hi,
> Thank you very much. Your reply make me clear with Kafka and cloud
> environment a lot. And it makes me remind that it seems
> advertised.host.name is available after Kafka 0.8. Perhaps we can't
> access Kafka on GCE from local machine before, is it correct?
>
> Best regards,
> James
>
>
>
> > Roger Hoover <ro...@gmail.com> 於 2014/6/14 上午12:32 寫道：
> >
> > I wouldn't say that Kafka's making it difficult. The cloud environment
> is
> > making it difficult. The VM that the Kafka broker is running on can only
> > see it's private IP (at the OS level) so you have to add the
> > advertised.host.name config so that it knows what public IP is assigned
> to
> > it.
> >
> >
> >> On Fri, Jun 13, 2014 at 8:32 AM, <ja...@yahoo.com.tw> wrote:
> >>
> >> Hi,
> >> I found this caused by broker's config. I miss typed
> advertise.host.name.
> >> Actually it should be advertised.host.name.
> >>
> >> However, I still don't know what's different between host.name and
> >> advertised.host.name. Why Kafka makes it so difficult to connect from
> >> public ip. Thanks your reply.
> >>
> >> Best regards,
> >> James
> >>
> >>
> >>
> >>
> >>> Jun Rao <ju...@gmail.com> 於 2014/6/13 下午10:22 寫道：
> >>>
> >>> Could you telnet to the public ip from your local machine?
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>>
> >>>> On Thu, Jun 12, 2014 at 4:04 AM, 傅駿浩 <ja...@yahoo.com.tw> wrote:
> >>>>
> >>>> Hi all,
> >>>>
> >>>> I want to use the KafkaSpout(where is
> >>
> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka
> >> )
> >>>> to read data from kafka. But my kafka and zookeeper cluster are
> builted
> >> on
> >>>> GCE(google cloud engine) and my storm testing is on local machine. The
> >>>> problem is I cannot set public IP by kafka config - server.properties
> to
> >>>> reach from my local(out of kafka's domain) storm machines.
> >>>> ***My server.properties is as follows:
> >>>> host.name=[Brokers's Private IP]
> >>>> advertise.port=9092
> >>>> advertise.host.name=[Brokers's Public IP]
> >>>>
> >>>> ***My KafkaSpout's config in Java is as follows:
> >>>> BrokerHosts a = new ZkHosts("[IP1],[IP2],[IP3]","/brokers");
> >>>> SpoutConfig kafkaConf = new SpoutConfig(a, topic, "/stormkafka",
> >>>> topic);//Using [topic] as id
> >>>> kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
> >>>> kafkaConf.zkServers = new ArrayList<String>() {
> >>>> {
> >>>> add("[IP1]");
> >>>> add("[IP1]");
> >>>> add("[IP1]");
> >>>> }
> >>>> };
> >>>> kafkaConf.zkPort = 2181;
> >>>> kafkaConf.forceFromStart = true;
> >>>> ps. I have 3-node kafka cluster and 3-node zookeeper cluster on 3
> >>>> machines. That's to say, Each machine with a broker and zookeeper node
> >>>>
> >>>> ======================================
> >>>> So the problem is it seens not to connect from the domain different
> from
> >>>> kafka's domain, e.g.my local storm machine. This is a big problem if
> I
> >>>> need to communcate with kafka out of its domain. The error exception
> is
> >> as
> >>>> follows, and can anyone help me with this issue. How can I config it
> in
> >> a
> >>>> right way so that I can subscribe topic of kafka from my local
> >>>> machine(storm).
> >>>> Thanks a lot :)
> >>>>
> >>>>
> >>>>
> >>>> java.lang.RuntimeException: java.net.NoRouteToHostException:
> >> 找不到通往該主機的路由路徑
> >>>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:83)
> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >>>> at
> >> storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45)
> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >>>> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:118)
> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >>>> at
> >>
> backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116$fn__5145.invoke(executor.clj:562)
> >>>> ~[na:na]
> >>>> at backtype.storm.util$async_loop$fn__390.invoke(util.clj:433)
> ~[na:na]
> >>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> >>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> >>>> Caused by: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
> >>>> at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_60]
> >>>> at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_60]
> >>>> at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_60]
> >>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> >>>> ~[na:1.7.0_60]
> >>>> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
> >>>> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
> >>>> at
> >>
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142)
> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
> >>>> at
> >>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
> >>>> at
> >>>>
> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:124)
> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
> >>>> at
> >>
> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
> >>>> ~[kafka_2.10-0.8.1.1.jar:na]
> >>>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:55)
> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >>>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:45)
> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >>>> at storm.kafka.PartitionManager.<init>(PartitionManager.java:77)
> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >>>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78)
> >>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >>>> ... 6 common frames omitted
> >>>>
> >>>> Best regards,
> >>>> James Fu
> >>
>
Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine
Posted by ja...@yahoo.com.tw.
Hi,
Thank you very much. Your reply make me clear with Kafka and cloud environment a lot. And it makes me remind that it seems advertised.host.name is available after Kafka 0.8. Perhaps we can't access Kafka on GCE from local machine before, is it correct?
Best regards,
James
> Roger Hoover <ro...@gmail.com> 於 2014/6/14 上午12:32 寫道：
>
> I wouldn't say that Kafka's making it difficult. The cloud environment is
> making it difficult. The VM that the Kafka broker is running on can only
> see it's private IP (at the OS level) so you have to add the
> advertised.host.name config so that it knows what public IP is assigned to
> it.
>
>
>> On Fri, Jun 13, 2014 at 8:32 AM, <ja...@yahoo.com.tw> wrote:
>>
>> Hi,
>> I found this caused by broker's config. I miss typed advertise.host.name.
>> Actually it should be advertised.host.name.
>>
>> However, I still don't know what's different between host.name and
>> advertised.host.name. Why Kafka makes it so difficult to connect from
>> public ip. Thanks your reply.
>>
>> Best regards,
>> James
>>
>>
>>
>>
>>> Jun Rao <ju...@gmail.com> 於 2014/6/13 下午10:22 寫道：
>>>
>>> Could you telnet to the public ip from your local machine?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>>> On Thu, Jun 12, 2014 at 4:04 AM, 傅駿浩 <ja...@yahoo.com.tw> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I want to use the KafkaSpout(where is
>> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka
>> )
>>>> to read data from kafka. But my kafka and zookeeper cluster are builted
>> on
>>>> GCE(google cloud engine) and my storm testing is on local machine. The
>>>> problem is I cannot set public IP by kafka config - server.properties to
>>>> reach from my local(out of kafka's domain) storm machines.
>>>> ***My server.properties is as follows:
>>>> host.name=[Brokers's Private IP]
>>>> advertise.port=9092
>>>> advertise.host.name=[Brokers's Public IP]
>>>>
>>>> ***My KafkaSpout's config in Java is as follows:
>>>> BrokerHosts a = new ZkHosts("[IP1],[IP2],[IP3]","/brokers");
>>>> SpoutConfig kafkaConf = new SpoutConfig(a, topic, "/stormkafka",
>>>> topic);//Using [topic] as id
>>>> kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>>>> kafkaConf.zkServers = new ArrayList<String>() {
>>>> {
>>>> add("[IP1]");
>>>> add("[IP1]");
>>>> add("[IP1]");
>>>> }
>>>> };
>>>> kafkaConf.zkPort = 2181;
>>>> kafkaConf.forceFromStart = true;
>>>> ps. I have 3-node kafka cluster and 3-node zookeeper cluster on 3
>>>> machines. That's to say, Each machine with a broker and zookeeper node
>>>>
>>>> ======================================
>>>> So the problem is it seens not to connect from the domain different from
>>>> kafka's domain, e.g.my local storm machine. This is a big problem if I
>>>> need to communcate with kafka out of its domain. The error exception is
>> as
>>>> follows, and can anyone help me with this issue. How can I config it in
>> a
>>>> right way so that I can subscribe topic of kafka from my local
>>>> machine(storm).
>>>> Thanks a lot :)
>>>>
>>>>
>>>>
>>>> java.lang.RuntimeException: java.net.NoRouteToHostException:
>> 找不到通往該主機的路由路徑
>>>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:83)
>>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>>>> at
>> storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45)
>>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>>>> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:118)
>>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>>>> at
>> backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116$fn__5145.invoke(executor.clj:562)
>>>> ~[na:na]
>>>> at backtype.storm.util$async_loop$fn__390.invoke(util.clj:433) ~[na:na]
>>>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>>>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
>>>> Caused by: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
>>>> at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_60]
>>>> at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_60]
>>>> at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_60]
>>>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
>>>> ~[na:1.7.0_60]
>>>> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>>>> ~[kafka_2.10-0.8.1.1.jar:na]
>>>> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
>>>> ~[kafka_2.10-0.8.1.1.jar:na]
>>>> at
>> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142)
>>>> ~[kafka_2.10-0.8.1.1.jar:na]
>>>> at
>> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
>>>> ~[kafka_2.10-0.8.1.1.jar:na]
>>>> at
>>>> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:124)
>>>> ~[kafka_2.10-0.8.1.1.jar:na]
>>>> at
>> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
>>>> ~[kafka_2.10-0.8.1.1.jar:na]
>>>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:55)
>>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>>>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:45)
>>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>>>> at storm.kafka.PartitionManager.<init>(PartitionManager.java:77)
>>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>>>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78)
>>>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>>>> ... 6 common frames omitted
>>>>
>>>> Best regards,
>>>> James Fu
>>
Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine
Posted by Roger Hoover <ro...@gmail.com>.
I wouldn't say that Kafka's making it difficult. The cloud environment is
making it difficult. The VM that the Kafka broker is running on can only
see it's private IP (at the OS level) so you have to add the
advertised.host.name config so that it knows what public IP is assigned to
it.
On Fri, Jun 13, 2014 at 8:32 AM, <ja...@yahoo.com.tw> wrote:
> Hi,
> I found this caused by broker's config. I miss typed advertise.host.name.
> Actually it should be advertised.host.name.
>
> However, I still don't know what's different between host.name and
> advertised.host.name. Why Kafka makes it so difficult to connect from
> public ip. Thanks your reply.
>
> Best regards,
> James
>
>
>
>
> > Jun Rao <ju...@gmail.com> 於 2014/6/13 下午10:22 寫道：
> >
> > Could you telnet to the public ip from your local machine?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >> On Thu, Jun 12, 2014 at 4:04 AM, 傅駿浩 <ja...@yahoo.com.tw> wrote:
> >>
> >> Hi all,
> >>
> >> I want to use the KafkaSpout(where is
> >>
> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka
> )
> >> to read data from kafka. But my kafka and zookeeper cluster are builted
> on
> >> GCE(google cloud engine) and my storm testing is on local machine. The
> >> problem is I cannot set public IP by kafka config - server.properties to
> >> reach from my local(out of kafka's domain) storm machines.
> >> ***My server.properties is as follows:
> >> host.name=[Brokers's Private IP]
> >> advertise.port=9092
> >> advertise.host.name=[Brokers's Public IP]
> >>
> >> ***My KafkaSpout's config in Java is as follows:
> >> BrokerHosts a = new ZkHosts("[IP1],[IP2],[IP3]","/brokers");
> >> SpoutConfig kafkaConf = new SpoutConfig(a, topic, "/stormkafka",
> >> topic);//Using [topic] as id
> >> kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
> >> kafkaConf.zkServers = new ArrayList<String>() {
> >> {
> >> add("[IP1]");
> >> add("[IP1]");
> >> add("[IP1]");
> >> }
> >> };
> >> kafkaConf.zkPort = 2181;
> >> kafkaConf.forceFromStart = true;
> >> ps. I have 3-node kafka cluster and 3-node zookeeper cluster on 3
> >> machines. That's to say, Each machine with a broker and zookeeper node
> >>
> >> ======================================
> >> So the problem is it seens not to connect from the domain different from
> >> kafka's domain, e.g.my local storm machine. This is a big problem if I
> >> need to communcate with kafka out of its domain. The error exception is
> as
> >> follows, and can anyone help me with this issue. How can I config it in
> a
> >> right way so that I can subscribe topic of kafka from my local
> >> machine(storm).
> >> Thanks a lot :)
> >>
> >>
> >>
> >> java.lang.RuntimeException: java.net.NoRouteToHostException:
> 找不到通往該主機的路由路徑
> >> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:83)
> >> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >> at
> storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45)
> >> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:118)
> >> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >> at
> >>
> backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116$fn__5145.invoke(executor.clj:562)
> >> ~[na:na]
> >> at backtype.storm.util$async_loop$fn__390.invoke(util.clj:433) ~[na:na]
> >> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> >> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> >> Caused by: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
> >> at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_60]
> >> at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_60]
> >> at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_60]
> >> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> >> ~[na:1.7.0_60]
> >> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> >> ~[kafka_2.10-0.8.1.1.jar:na]
> >> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
> >> ~[kafka_2.10-0.8.1.1.jar:na]
> >> at
> >>
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142)
> >> ~[kafka_2.10-0.8.1.1.jar:na]
> >> at
> >>
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
> >> ~[kafka_2.10-0.8.1.1.jar:na]
> >> at
> >> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:124)
> >> ~[kafka_2.10-0.8.1.1.jar:na]
> >> at
> >>
> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
> >> ~[kafka_2.10-0.8.1.1.jar:na]
> >> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:55)
> >> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:45)
> >> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >> at storm.kafka.PartitionManager.<init>(PartitionManager.java:77)
> >> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78)
> >> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> >> ... 6 common frames omitted
> >>
> >> Best regards,
> >> James Fu
>
Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine
Posted by ja...@yahoo.com.tw.
Hi,
I found this caused by broker's config. I miss typed advertise.host.name. Actually it should be advertised.host.name.
However, I still don't know what's different between host.name and advertised.host.name. Why Kafka makes it so difficult to connect from public ip. Thanks your reply.
Best regards,
James
> Jun Rao <ju...@gmail.com> 於 2014/6/13 下午10:22 寫道：
>
> Could you telnet to the public ip from your local machine?
>
> Thanks,
>
> Jun
>
>
>> On Thu, Jun 12, 2014 at 4:04 AM, 傅駿浩 <ja...@yahoo.com.tw> wrote:
>>
>> Hi all,
>>
>> I want to use the KafkaSpout(where is
>> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka)
>> to read data from kafka. But my kafka and zookeeper cluster are builted on
>> GCE(google cloud engine) and my storm testing is on local machine. The
>> problem is I cannot set public IP by kafka config - server.properties to
>> reach from my local(out of kafka's domain) storm machines.
>> ***My server.properties is as follows:
>> host.name=[Brokers's Private IP]
>> advertise.port=9092
>> advertise.host.name=[Brokers's Public IP]
>>
>> ***My KafkaSpout's config in Java is as follows:
>> BrokerHosts a = new ZkHosts("[IP1],[IP2],[IP3]","/brokers");
>> SpoutConfig kafkaConf = new SpoutConfig(a, topic, "/stormkafka",
>> topic);//Using [topic] as id
>> kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>> kafkaConf.zkServers = new ArrayList<String>() {
>> {
>> add("[IP1]");
>> add("[IP1]");
>> add("[IP1]");
>> }
>> };
>> kafkaConf.zkPort = 2181;
>> kafkaConf.forceFromStart = true;
>> ps. I have 3-node kafka cluster and 3-node zookeeper cluster on 3
>> machines. That's to say, Each machine with a broker and zookeeper node
>>
>> ======================================
>> So the problem is it seens not to connect from the domain different from
>> kafka's domain, e.g.my local storm machine. This is a big problem if I
>> need to communcate with kafka out of its domain. The error exception is as
>> follows, and can anyone help me with this issue. How can I config it in a
>> right way so that I can subscribe topic of kafka from my local
>> machine(storm).
>> Thanks a lot :)
>>
>>
>>
>> java.lang.RuntimeException: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:83)
>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45)
>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:118)
>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> at
>> backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116$fn__5145.invoke(executor.clj:562)
>> ~[na:na]
>> at backtype.storm.util$async_loop$fn__390.invoke(util.clj:433) ~[na:na]
>> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
>> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
>> Caused by: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
>> at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_60]
>> at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_60]
>> at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_60]
>> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
>> ~[na:1.7.0_60]
>> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
>> ~[kafka_2.10-0.8.1.1.jar:na]
>> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
>> ~[kafka_2.10-0.8.1.1.jar:na]
>> at
>> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142)
>> ~[kafka_2.10-0.8.1.1.jar:na]
>> at
>> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
>> ~[kafka_2.10-0.8.1.1.jar:na]
>> at
>> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:124)
>> ~[kafka_2.10-0.8.1.1.jar:na]
>> at
>> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
>> ~[kafka_2.10-0.8.1.1.jar:na]
>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:55)
>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:45)
>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> at storm.kafka.PartitionManager.<init>(PartitionManager.java:77)
>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78)
>> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
>> ... 6 common frames omitted
>>
>> Best regards,
>> James Fu
Re: [Issue] Cannot connect to Kafka cluster on GCE from local machine
Posted by Jun Rao <ju...@gmail.com>.
Could you telnet to the public ip from your local machine?
Thanks,
Jun
On Thu, Jun 12, 2014 at 4:04 AM, 傅駿浩 <ja...@yahoo.com.tw> wrote:
> Hi all,
>
> I want to use the KafkaSpout(where is
> https://github.com/apache/incubator-storm/tree/master/external/storm-kafka)
> to read data from kafka. But my kafka and zookeeper cluster are builted on
> GCE(google cloud engine) and my storm testing is on local machine. The
> problem is I cannot set public IP by kafka config - server.properties to
> reach from my local(out of kafka's domain) storm machines.
> ***My server.properties is as follows:
> host.name=[Brokers's Private IP]
> advertise.port=9092
> advertise.host.name=[Brokers's Public IP]
>
> ***My KafkaSpout's config in Java is as follows:
> BrokerHosts a = new ZkHosts("[IP1],[IP2],[IP3]","/brokers");
> SpoutConfig kafkaConf = new SpoutConfig(a, topic, "/stormkafka",
> topic);//Using [topic] as id
> kafkaConf.scheme = new SchemeAsMultiScheme(new StringScheme());
> kafkaConf.zkServers = new ArrayList<String>() {
> {
> add("[IP1]");
> add("[IP1]");
> add("[IP1]");
> }
> };
> kafkaConf.zkPort = 2181;
> kafkaConf.forceFromStart = true;
> ps. I have 3-node kafka cluster and 3-node zookeeper cluster on 3
> machines. That's to say, Each machine with a broker and zookeeper node
>
> ======================================
> So the problem is it seens not to connect from the domain different from
> kafka's domain, e.g.my local storm machine. This is a big problem if I
> need to communcate with kafka out of its domain. The error exception is as
> follows, and can anyone help me with this issue. How can I config it in a
> right way so that I can subscribe topic of kafka from my local
> machine(storm).
> Thanks a lot :)
>
>
>
> java.lang.RuntimeException: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:83)
> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> at storm.kafka.ZkCoordinator.getMyManagedPartitions(ZkCoordinator.java:45)
> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> at storm.kafka.KafkaSpout.nextTuple(KafkaSpout.java:118)
> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> at
> backtype.storm.daemon.executor$eval5100$fn__5101$fn__5116$fn__5145.invoke(executor.clj:562)
> ~[na:na]
> at backtype.storm.util$async_loop$fn__390.invoke(util.clj:433) ~[na:na]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> Caused by: java.net.NoRouteToHostException: 找不到通往該主機的路由路徑
> at sun.nio.ch.Net.connect0(Native Method) ~[na:1.7.0_60]
> at sun.nio.ch.Net.connect(Net.java:465) ~[na:1.7.0_60]
> at sun.nio.ch.Net.connect(Net.java:457) ~[na:1.7.0_60]
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:670)
> ~[na:1.7.0_60]
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> ~[kafka_2.10-0.8.1.1.jar:na]
> at kafka.consumer.SimpleConsumer.connect(SimpleConsumer.scala:44)
> ~[kafka_2.10-0.8.1.1.jar:na]
> at
> kafka.consumer.SimpleConsumer.getOrMakeConnection(SimpleConsumer.scala:142)
> ~[kafka_2.10-0.8.1.1.jar:na]
> at
> kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:69)
> ~[kafka_2.10-0.8.1.1.jar:na]
> at
> kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:124)
> ~[kafka_2.10-0.8.1.1.jar:na]
> at
> kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
> ~[kafka_2.10-0.8.1.1.jar:na]
> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:55)
> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> at storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:45)
> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> at storm.kafka.PartitionManager.<init>(PartitionManager.java:77)
> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> at storm.kafka.ZkCoordinator.refresh(ZkCoordinator.java:78)
> ~[storm-kafka-0.8-plus-0.4.0.jar:na]
> ... 6 common frames omitted
>
> Best regards,
> James Fu