You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com> on 2014/02/04 10:54:22 UTC
RE: Reg Exception in Kafka
I downloaded from the trunk and set up the properties
host.name= <local IP>
advertise.host.name=<public IP>
Even after this change, I get the ConnectException. The details logs are given below. Is there any workaround for this?
[ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from 54.241.44.129:9094
[DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched metadata for 1 topic(s) Set(mytopic)
[DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition info for topic mytopic
[DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has leader 3
[DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions registered for topic: mytopic are 0
[DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no compression to [mytopic,0]
[DEBUG] [main 2014-02-04 17:46:01,833] Producer sending messages with correlation id 2 for topics [mytopic,0] to broker 3 on 10.199.31.87:9094
[ERROR] [main 2014-02-04 17:46:22,850] Producer connection to 10.199.31.87:9094 unsuccessful
java.net.ConnectException: Connection timed out: connect
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:254)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
at scala.collection.Iterator$class.foreach(Iterator.scala:631)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
at com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
[ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer request with correlation id 2 to broker 3 with data for partitions [mytopic,0]
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Wednesday, January 29, 2014 11:12 PM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
Hmm, it's weird that EC2 only allows you to bind to local ip. Could some
EC2 users here help out?
Also, we recently added https://issues.apache.org/jira/browse/KAFKA-1092,
which allows one to use a different ip for binding and connecting. You can see if this works for you. The patch is only in trunk though.
Thanks,
Jun
On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> I don't think so. I forgot to include the ifconfig output. Actually
> the public IP is not one of the IP configured in the Ethernet interfaces.
> Only the Local IP is configured in eth0.
> Is there any solution to this?
>
> Ifconfig O/P:
>
> eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> inet addr:10.X.X.X Bcast:10.199.31.127 Mask:255.255.255.192
> inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:1000
> RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> Interrupt:25
>
> lo Link encap:Local Loopback
> inet addr:127.0.0.1 Mask:255.0.0.0
> inet6 addr: ::1/128 Scope:Host
> UP LOOPBACK RUNNING MTU:16436 Metric:1
> RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> collisions:0 txqueuelen:0
> RX bytes:109133672 (104.0 MiB) TX bytes:109133672 (104.0
> MiB)
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, January 29, 2014 12:27 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Could it be a port conflict?
>
> Thanks,
>
> Jun
>
>
> On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
>
> > Jun,
> >
> > Thanks for your help.
> > I get the following exception :
> > kafka.common.KafkaException: Socket server failed to bind to
> > 54.241.44.129:9092: Cannot assign requested address.
> > at
> kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > at
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > at kafka.Kafka$.main(Kafka.scala:46)
> > at kafka.Kafka.main(Kafka.scala) Caused by:
> > java.net.BindException: Cannot assign requested address
> > at sun.nio.ch.Net.bind0(Native Method)
> > at sun.nio.ch.Net.bind(Net.java:174)
> > at
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > at
> kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > ... 6 more
> >
> > The entire stack trace of the logs are placed below.
> >
> > [2014-01-29 01:18:23,136] INFO Verifying properties
> > (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,176] INFO Property host.name is overridden to
> > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,177] INFO Property port is overridden to 9092
> > (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,177] INFO Property socket.request.max.bytes is
> > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,177] INFO Property num.io.threads is overridden
> > to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden to
> > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,178] INFO Property log.cleanup.interval.mins is
> > overridden to 1 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,178] INFO Property socket.send.buffer.bytes is
> > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms is
> > overridden to 1000 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > overridden to
> > localhost:2181 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,180] INFO Property broker.id is overridden to 1
> > (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > overridden to 168 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > overridden to 2 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,180] INFO Property socket.receive.buffer.bytes
> > is overridden to 1048576 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,181] INFO Property
> > zookeeper.connection.timeout.msis overridden to 1000000
> > (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,181] INFO Property num.partitions is overridden
> > to 2
> > (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,181] INFO Property log.flush.interval.messages
> > is overridden to 10000 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > overridden to
> > 536870912 (kafka.utils.VerifiableProperties)
> > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > (kafka.server.KafkaServer)
> > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1] Starting
> > log cleaner every 60000 ms (kafka.log.LogManager)
> > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1] Starting
> > log flusher every 3000 ms with the following overrides Map()
> > (kafka.log.LogManager)
> > [2014-01-29 01:18:23,330] FATAL Fatal error during KafkaServerStable
> > startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> > kafka.common.KafkaException: Socket server failed to bind to
> > 54.241.44.129:9092: Cannot assign requested address.
> > at
> kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > at
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > at kafka.Kafka$.main(Kafka.scala:46)
> > at kafka.Kafka.main(Kafka.scala) Caused by:
> > java.net.BindException: Cannot assign requested address
> > at sun.nio.ch.Net.bind0(Native Method)
> > at sun.nio.ch.Net.bind(Net.java:174)
> > at
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > at
> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > at
> kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > ... 6 more
> > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > (kafka.server.KafkaServer)
> > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1], Shutting
> > down
> > (kafka.network.SocketServer)
> > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1], Shutdown
> > completed (kafka.network.SocketServer)
> > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > (kafka.utils.KafkaScheduler)
> > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down completed
> > (kafka.server.KafkaServer)
> > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > (kafka.server.KafkaServer)
> >
> > Regards
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Tuesday, January 28, 2014 11:30 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > You should should use the public IP for host.name. What's the error
> > you see during broker startup?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I checked the faq. I did change the host.name in server properties.
> > > After changing it I get ConnectException.
> > >
> > > The problem here is in EC2 have a different public IP address
> > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > I set the host.name property to the local IP address which is
> > > 10.x.x.x. I think because of this there is a ConnectException.
> > > When I set the host.name to the public ip address (55.x.x.x), I
> > > cannot even start the broker.
> > >
> > > What should be the IP address that is to be given in host.nameproperty.
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Have you looked at
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > ?
> > >
> > > thanks,
> > >
> > > Jun
> > >
> > >
> > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > There are
> > > > 3 brokers each running in the port 9092,9093,9094. The zookeeper
> > > > is running in the port 2181.
> > > > When I send message to the brokers from my PC, I get an
> > > > exception which is given below. I did a dump in the remote
> > > > server, the request is received in the remote server.
> > > > I am able to locally test the consumer/producer script present
> > > > in the bin folder. What am I missing? Can you kindly help me in
> > > > this
> > error?
> > > > Any help will be highly grateful.
> > > >
> > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying properties [
> > > > INFO] [main 2014-01-27 16:06:50,108] Property
> > > > metadata.broker.list is overridden to
> > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > 2014-01-27 16:06:50,108] Property serializer.class is overridden
> > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > 16:06:50,154]
> > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > id:0,host:54.241.44.129,port:9093 with correlation id 0 for 1
> > > > topic(s)
> > > > Set(mytopic)
> > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > SO_TIMEOUT =
> > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > SO_SNDBUF =
> > > > 102400 (requested 102400).
> > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > 15:59:44,018] Disconnecting from
> > > > 54.241.44.129:9093
> > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully fetched
> > > > metadata for 1
> > > > topic(s) Set(mytopic)
> > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker partition
> > > > info for topic mytopic [DEBUG] [main 2014-01-27 15:59:44,060]
> > > > Partition [mytopic,0] has leader 2 [DEBUG] [main 2014-01-27
> > > > 15:59:44,072] Broker partitions registered for
> > > > topic: mytopic are 0
> > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages with
> > > > no compression to [mytopic,0] [DEBUG] [main 2014-01-27
> > > > 15:59:44,109] Producer sending messages with correlation id 2
> > > > for topics [mytopic,0] to broker 2 on
> > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection to
> > > > ip-10-199-31-87.us-west-1.compute.internal:9093 unsuccessful
> > > > java.nio.channels.UnresolvedAddressException
> > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
> > > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > at
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> )
> > > > at
> > > >
> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > nc
> > > Pr
> > > oducer.scala:68)
> > > > at
> > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > ap
> > > ply$mcV$sp(SyncProducer.scala:102)
> > > > at
> > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > ap
> > > ply(SyncProducer.scala:102)
> > > > at
> > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > ap
> > > ply(SyncProducer.scala:102)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > scala:101)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > :1
> > 01)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > :1
> > 01)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > ul
> > > tE
> > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > ed
> > > Da
> > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > ed
> > > Da
> > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > at
> > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > sc
> > > al
> > > a:80)
> > > > at
> > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > sc
> > > al
> > > a:80)
> > > > at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > at
> > > >
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> )
> > > > at
> > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > 194)
> > > > at
> > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > fa
> > > ul
> > > tEventHandler.scala:100)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > sc
> > > ala:72)
> > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > at
> > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > at
> > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:21)
> > > > [ WARN] [main 2014-01-27 15:59:44,139] Failed to send producer
> > > > request with correlation id 2 to broker 2 with data for
> > > > partitions [mytopic,0]
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > >
> >
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
I haven't enabled the JMX. I disabled the configuration and started the Kafka brokers.
Thanks
Bala
-----Original Message-----
From: Joe Stein [mailto:joe.stein@stealth.ly]
Sent: Friday, 21 February, 2014 11:49 AM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
Have you changed the JMX port when starting?
On Thu, Feb 20, 2014 at 10:37 PM, Balasubramanian Jayaraman (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> All the brokers reside in the same server and are listening on the
> port 10092,10093,10094. From the same machine I can connect to the
> zookeeper and the brokers. But When I tried to connect from an
> external machine (from Eclipse), I get an exception as communicated
> earlier. I was not able to connect to any of the brokers. I get the
> same exception while connecting to all the brokers.
>
> Regards
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, 20 February, 2014 12:05 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Can you make the connection from the controller host to the other broker?
> Also, what's the # open file handlers on each broker? Do you see any
> "too many open file handler" error in the broker log?
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman
> (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I don't think so. I am able to connect to the server using a putty.
> > It is VM running in Amazon cloud.
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, 19 February, 2014 12:32 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Any issue with the network?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I just came from vacation.
> > > When I tested it, I get a Connect Exception. The complete stack
> > > trace is given below.
> > >
> > > Exception Stack Trace
> > >
> > > Java Thin Client:
> > >
> > > ava.net.ConnectException: Connection timed out: connect
> > > at sun.nio.ch.Net.connect0(Native Method)
> > > at sun.nio.ch.Net.connect(Net.java:465)
> > > at sun.nio.ch.Net.connect(Net.java:457)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > at
> > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sync
> > Pr
> > oducer.scala:68)
> > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > at
> > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > at
> > >
> > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.sc
> > al
> > a:82)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV
> > $s
> > p(DefaultEventHandler.scala:67)
> > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > sc
> > ala:67)
> > > at kafka.producer.Producer.send(Producer.scala:76)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at
> > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > java:48)
> > > at
> > >
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(K
> > af
> > kaProducerTest.java:33)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
> > .j
> > ava:57)
> > > at
> > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> > ss
> > orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > > at junit.framework.TestResult.run(TestResult.java:109)
> > > at junit.framework.TestCase.run(TestCase.java:118)
> > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run
> > (J
> > Unit3TestReference.java:131)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > java:38)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remo
> > te
> > TestRunner.java:467)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remo
> > te
> > TestRunner.java:683)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTes
> > tR
> > unner.java:390)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTe
> > st
> > Runner.java:197)
> > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata
> > > with correlation id 0 for topics [Set(mytopic)] from broker
> > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > >
> > > Kafka Server:
> > > [2014-02-19 03:38:22,587] ERROR
> > > [Controller-3-to-broker-3-send-thread],
> > > Controller 3's connection to broker
> > > id:3,host:54.xx.xx.xx,port:10094 was unsuccessful
> > > (kafka.controller.RequestSendThread)
> > > java.net.ConnectException: Connection timed out
> > > at sun.nio.ch.Net.connect(Native Method)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at
> > >
> > kafka.controller.RequestSendThread.connectToBroker(ControllerChannel
> > Ma
> > nager.scala:172)
> > > at
> > >
> > kafka.controller.RequestSendThread.<init>(ControllerChannelManager.s
> > ca
> > la:115)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.kafka$controller$Controlle
> > rC
> > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.addBroker(ControllerChanne
> > lM
> > anager.scala:65)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.s
> > ca
> > la:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.s
> > ca
> > la:265)
> > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChil
> > dC
> > hange(ReplicaStateMachine.scala:255)
> > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > at
> > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup
> > callback
> > > for 3 (kafka.controller.KafkaController)
> > > [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> > > Invoking state change to OnlineReplica for replicas
> > > (kafka.controller.ReplicaStateMachine)
> > > [2014-02-19 03:38:22,595] INFO
> > > [Controller-3-to-broker-3-send-thread],
> > > Starting (kafka.controller.RequestSendThread)
> > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> > > Broker change listener fired for path /brokers/ids with children
> > > 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> > > Newly added brokers: 2,1, deleted brokers: , all live brokers:
> > > 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > >
> > >
> > > -----Original Message-----
> > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > balasubramanian.jayaraman@autodesk.com]
> > > Sent: Thursday, February 06, 2014 5:40 PM
> > > To: users@kafka.apache.org
> > > Subject: RE: Reg Exception in Kafka
> > >
> > > After correcting the spelling, it is registered to correct public IP.
> > >
> > > Thanks
> > > Bala
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > I think you mis-spelled the property name. It should be "
> > > advertised.host.name", instead of advertise.host.name
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > Thanks Joel.
> > > >
> > > > It seems the Broker is registered to the zookeeper with the IP
> > > > 10.199.31.87. The output pf the command is given below.
> > > >
> > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181
> > > > get
> > > > /brokers/ids/1
> > > > Connecting to 0.0.0.0:2181
> > > >
> > > > WATCHER::
> > > >
> > > > WatchedEvent state:SyncConnected type:None path:null
> > > >
> > > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > vers
> > > > ion":1,"port":9092}
> > > > cZxid = 0x2d
> > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu
> > > > Feb
> > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength =
> > > > 89
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Use zookeeper-shell script:
> > > >
> > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > /brokers/ids/<your broker id>
> > > >
> > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > Jayaraman
> > > > (Contingent) wrote:
> > > > > Where should I look for these information. From the logs, I
> > > > > could see
> > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server
> > > > is started in port 9082 and bind to IP 10.x.x.x.
> > > > > If I don't give the host.name in server.properties, I get "
> > > > java.nio.channels.UnresolvedAddressException" and If I give the
> > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > It is the same behavior as in 0.8.0.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > http://10.199.31.87:9094/>)
> > > > > is still the local ip, not the public one. Could you check the
> > > > > broker registration in zk (
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+str
> > > > uc
> > > > ture
> > > > s+in+Zookeeper
> > > > )
> > > > > and see what's the host/port of the broker?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I downloaded from the trunk and set up the properties
> > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > >
> > > > > > Even after this change, I get the ConnectException. The
> > > > > > details logs
> > > > are
> > > > > > given below. Is there any workaround for this?
> > > > > >
> > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > 54.241.44.129:9094
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > > > metadata
> > > > for 1
> > > > > > topic(s) Set(mytopic)
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > partition info
> > > > for
> > > > > > topic mytopic
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0]
> > > > > > has
> > > > leader 3
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > registered for
> > > > > > topic: mytopic are 0
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages
> > > > > > with no compression to [mytopic,0] [DEBUG] [main 2014-02-04
> > > > > > 17:46:01,833] Producer sending messages with correlation id
> > > > > > 2 for topics [mytopic,0] to broker 3 on
> > > > 10.199.31.87:9094
> > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection
> > > > > > to
> > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > at
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala
> > > :1
> > > 61)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(
> > > > Sy
> > > > ncPr
> > > > oducer.scala:68)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > p$
> > > > 1.ap
> > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > p$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > p$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > scala:101)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la:1
> > > > 01)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la:1
> > > > 01)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$De
> > > > fa
> > > > ultE
> > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > at
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala
> > > :1
> > > 61)
> > > > > > at
> > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > 194)
> > > > > > at
> > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > at
> > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(
> > > > De
> > > > faul
> > > > tEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHand
> > > > le
> > > > r.sc
> > > > ala:72)
> > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > at
> > > > > >
> > > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > java:48)
> > > > > > at
> > > > > >
> > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProduc
> > > > er
> > > > (Kaf
> > > > kaProducerTest.java:33)
> > > > > > at
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> > > > > > at
> > > > > >
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor
> > > > Im
> > > > pl.j
> > > > ava:57)
> > > > > > at
> > > > > >
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
> > > > Ac
> > > > cess
> > > > orImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > at
> > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > at
> > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference
> > > > .r
> > > > un(J
> > > > Unit3TestReference.java:131)
> > > > > > at
> > > > > >
> > > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > java:38)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(
> > > > Re
> > > > mote
> > > > TestRunner.java:467)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(
> > > > Re
> > > > mote
> > > > TestRunner.java:683)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(Remot
> > > > eT
> > > > estR
> > > > unner.java:390)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remo
> > > > te
> > > > Test
> > > > Runner.java:197)
> > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send
> > > > > > producer request with correlation id 2 to broker 3 with data
> > > > > > for partitions [mytopic,0]
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > > Could
> > > > some
> > > > > > EC2 users here help out?
> > > > > >
> > > > > > Also, we recently added
> > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > which allows one to use a different ip for binding and
> connecting.
> > > > > > You
> > > > can
> > > > > > see if this works for you. The patch is only in trunk though.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > > > (Contingent) <
> > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > > Actually the public IP is not one of the IP configured in
> > > > > > > the Ethernet
> > > > interfaces.
> > > > > > > Only the Local IP is configured in eth0.
> > > > > > > Is there any solution to this?
> > > > > > >
> > > > > > > Ifconfig O/P:
> > > > > > >
> > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > Mask:255.255.255.192
> > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> frame:0
> > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > carrier:0
> > > > > > > collisions:0 txqueuelen:1000
> > > > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154
> > > > > > > (14.3
> > > MiB)
> > > > > > > Interrupt:25
> > > > > > >
> > > > > > > lo Link encap:Local Loopback
> > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > RX packets:1379711 errors:0 dropped:0 overruns:0
> > frame:0
> > > > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> > > carrier:0
> > > > > > > collisions:0 txqueuelen:0
> > > > > > > RX bytes:109133672 (104.0 MiB) TX
> > > > > > > bytes:109133672
> > > > > > > (104.0
> > > > > > > MiB)
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Could it be a port conflict?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > Thanks for your help.
> > > > > > > > I get the following exception :
> > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > bind to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.s
> > > > ca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > >
> > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > >
> > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > overridden to
> > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is
> > > > > > > > overridden to
> > > > > > > > 9092
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads
> > > > > > > > is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > overridden to
> > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > log.cleanup.interval.mins
> > > > is
> > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > zookeeper.connect is overridden to
> > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > overridden
> > > > to 1
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > num.network.threads is overridden to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions
> > > > > > > > is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,182] INFO Property
> > > > > > > > log.segment.bytes is overridden to
> > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1],
> > > > > > > > Starting
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log cleaner every 60000 ms
> > > > > > > > (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log flusher every 3000 ms with the following
> > > > > > > > overrides Map()
> > > > > > > > (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > KafkaServerStable
> > > > > > > > startup. Prepare to shutdown
> > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > bind to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.s
> > > > ca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1],
> > > > > > > > Shutting down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutting
> > > > > > > > down
> > > > > > > > (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutdown
> > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut
> > > > > > > > down
> > > > completed
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1],
> > > > > > > > Shutting down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > You should should use the public IP for host.name.
> > > > > > > > What's the
> > > > error
> > > > > > > > you see during broker startup?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian
> > > > > > > > Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > I checked the faq. I did change the host.name in
> > > > > > > > > server
> > > > properties.
> > > > > > > > > After changing it I get ConnectException.
> > > > > > > > >
> > > > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > > > address
> > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > which is 10.x.x.x. I think because of this there is a
> > > ConnectException.
> > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > >
> > > > > > > > > What should be the IP address that is to be given in
> > > > > > host.nameproperty.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > Have you looked at
> > > > > > > > >
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,
> > > > wh
> > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > ?
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have a remote server (EC2) setup with Kafka
> > > > > > > > > > cluster
> > setup.
> > > > > > > > > > There are
> > > > > > > > > > 3 brokers each running in the port 9092,9093,9094.
> > > > > > > > > > The
> > > > zookeeper
> > > > > > > > > > is running in the port 2181.
> > > > > > > > > > When I send message to the brokers from my PC, I get
> > > > > > > > > > an exception which is given below. I did a dump in
> > > > > > > > > > the remote server, the request is received in the
> > > > > > > > > > remote
> server.
> > > > > > > > > > I am able to locally test the consumer/producer
> > > > > > > > > > script present in the bin folder. What am I missing?
> > > > > > > > > > Can you kindly help me in this
> > > > > > > > error?
> > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > >
> > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:
> > > > > > > > > > 90
> > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > request.required.acks is overridden to 1 [ INFO]
> > > > > > > > > > [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property
> > > > > > > > > > key.serializer.class is overridden to
> > > > > > > > > > kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class
> > > > > > > > > > is
> > > > overridden
> > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,154]
> > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from
> > > > > > > > > > broker
> > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation
> > > > > > > > > > id
> > > > > > > > > > 0 for 1
> > > > > > > > > > topic(s)
> > > > > > > > > > Set(mytopic)
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created
> > > > > > > > > > socket with SO_TIMEOUT =
> > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested
> > > > > > > > > > -1), SO_SNDBUF =
> > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > > > 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > 54.241.44.129:9093
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > > > fetched metadata for 1
> > > > > > > > > > topic(s) Set(mytopic) [DEBUG] [main 2014-01-27
> > > > > > > > > > 15:59:44,058] Getting broker partition info for
> > > > > > > > > > topic mytopic [DEBUG] [main
> > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072]
> > > > > > > > > > Broker partitions registered for
> > > > > > > > > > topic: mytopic are 0 [DEBUG] [main 2014-01-27
> > > > > > > > > > 15:59:44,091] Sending 1 messages with no compression
> > > > > > > > > > to [mytopic,0] [DEBUG] [main
> > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > with correlation id 2 for topics [mytopic,0] to
> > > > > > > > > > broker
> > > > > > > > > > 2 on
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > connection to
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > at
> > > > > > > > >
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > 640)
> > > > > > > > > > at
> > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.sc
> > > > > > > > al
> > > > > > > > a:57
> > > > > > > > )
> > > > > > > > > > at
> > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.sca
> > > > la
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(
> > > > Sy
> > > > > > > > > nc
> > > > > > > > > Pr
> > > > > > > > > oducer.scala:68)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > scala:101)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$De
> > > > fa
> > > > > > > > > ul
> > > > > > > > > tE
> > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.sca
> > > > la
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > 194)
> > > > > > > > > > at
> > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.sc
> > > > > > > > al
> > > > > > > > a:39
> > > > > > > > )
> > > > > > > > > > at
> > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(
> > > > De
> > > > > > > > > fa
> > > > > > > > > ul
> > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > sc
> > > > > > > > > ala:72)
> > > > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > at
> > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:3
> > > > 9)
> > > > > > > > > > at
> > > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.
> > > > > > > > > > ja
> > > > > > > > > > va:2
> > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to
> > > > > > > > > > send producer request with correlation id 2 to
> > > > > > > > > > broker
> > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>
Re: Reg Exception in Kafka
Posted by Joe Stein <jo...@stealth.ly>.
Have you changed the JMX port when starting?
On Thu, Feb 20, 2014 at 10:37 PM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> All the brokers reside in the same server and are listening on the port
> 10092,10093,10094. From the same machine I can connect to the zookeeper and
> the brokers. But When I tried to connect from an external machine (from
> Eclipse), I get an exception as communicated earlier. I was not able to
> connect to any of the brokers. I get the same exception while connecting to
> all the brokers.
>
> Regards
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, 20 February, 2014 12:05 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Can you make the connection from the controller host to the other broker?
> Also, what's the # open file handlers on each broker? Do you see any "too
> many open file handler" error in the broker log?
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I don't think so. I am able to connect to the server using a putty. It
> > is VM running in Amazon cloud.
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, 19 February, 2014 12:32 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Any issue with the network?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I just came from vacation.
> > > When I tested it, I get a Connect Exception. The complete stack
> > > trace is given below.
> > >
> > > Exception Stack Trace
> > >
> > > Java Thin Client:
> > >
> > > ava.net.ConnectException: Connection timed out: connect
> > > at sun.nio.ch.Net.connect0(Native Method)
> > > at sun.nio.ch.Net.connect(Net.java:465)
> > > at sun.nio.ch.Net.connect(Net.java:457)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > at
> > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> > oducer.scala:68)
> > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > at
> > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > at
> > >
> > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scal
> > a:82)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$s
> > p(DefaultEventHandler.scala:67)
> > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> > ala:67)
> > > at kafka.producer.Producer.send(Producer.scala:76)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at
> > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > java:48)
> > > at
> > >
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> > kaProducerTest.java:33)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:57)
> > > at
> > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > > at junit.framework.TestResult.run(TestResult.java:109)
> > > at junit.framework.TestCase.run(TestCase.java:118)
> > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> > Unit3TestReference.java:131)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > java:38)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:467)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:683)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> > unner.java:390)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> > Runner.java:197)
> > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata with
> > > correlation id 0 for topics [Set(mytopic)] from broker
> > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > >
> > > Kafka Server:
> > > [2014-02-19 03:38:22,587] ERROR
> > > [Controller-3-to-broker-3-send-thread],
> > > Controller 3's connection to broker id:3,host:54.xx.xx.xx,port:10094
> > > was unsuccessful (kafka.controller.RequestSendThread)
> > > java.net.ConnectException: Connection timed out
> > > at sun.nio.ch.Net.connect(Native Method)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at
> > >
> > kafka.controller.RequestSendThread.connectToBroker(ControllerChannelMa
> > nager.scala:172)
> > > at
> > >
> > kafka.controller.RequestSendThread.<init>(ControllerChannelManager.sca
> > la:115)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.kafka$controller$ControllerC
> > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.addBroker(ControllerChannelM
> > anager.scala:65)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.sca
> > la:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.sca
> > la:265)
> > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildC
> > hange(ReplicaStateMachine.scala:255)
> > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > at
> > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup
> > callback
> > > for 3 (kafka.controller.KafkaController)
> > > [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> > > Invoking state change to OnlineReplica for replicas
> > > (kafka.controller.ReplicaStateMachine)
> > > [2014-02-19 03:38:22,595] INFO
> > > [Controller-3-to-broker-3-send-thread],
> > > Starting (kafka.controller.RequestSendThread)
> > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> > > Broker change listener fired for path /brokers/ids with children
> > > 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> > > Newly added brokers: 2,1, deleted brokers: , all live brokers: 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > >
> > >
> > > -----Original Message-----
> > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > balasubramanian.jayaraman@autodesk.com]
> > > Sent: Thursday, February 06, 2014 5:40 PM
> > > To: users@kafka.apache.org
> > > Subject: RE: Reg Exception in Kafka
> > >
> > > After correcting the spelling, it is registered to correct public IP.
> > >
> > > Thanks
> > > Bala
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > I think you mis-spelled the property name. It should be "
> > > advertised.host.name", instead of advertise.host.name
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > Thanks Joel.
> > > >
> > > > It seems the Broker is registered to the zookeeper with the IP
> > > > 10.199.31.87. The output pf the command is given below.
> > > >
> > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> > > > /brokers/ids/1
> > > > Connecting to 0.0.0.0:2181
> > > >
> > > > WATCHER::
> > > >
> > > > WatchedEvent state:SyncConnected type:None path:null
> > > >
> > > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > vers
> > > > ion":1,"port":9092}
> > > > cZxid = 0x2d
> > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu Feb
> > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength = 89
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Use zookeeper-shell script:
> > > >
> > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > /brokers/ids/<your broker id>
> > > >
> > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > Jayaraman
> > > > (Contingent) wrote:
> > > > > Where should I look for these information. From the logs, I
> > > > > could see
> > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server
> > > > is started in port 9082 and bind to IP 10.x.x.x.
> > > > > If I don't give the host.name in server.properties, I get "
> > > > java.nio.channels.UnresolvedAddressException" and If I give the
> > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > It is the same behavior as in 0.8.0.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > http://10.199.31.87:9094/>)
> > > > > is still the local ip, not the public one. Could you check the
> > > > > broker registration in zk (
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+struc
> > > > ture
> > > > s+in+Zookeeper
> > > > )
> > > > > and see what's the host/port of the broker?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I downloaded from the trunk and set up the properties
> > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > >
> > > > > > Even after this change, I get the ConnectException. The
> > > > > > details logs
> > > > are
> > > > > > given below. Is there any workaround for this?
> > > > > >
> > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > 54.241.44.129:9094
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > > > metadata
> > > > for 1
> > > > > > topic(s) Set(mytopic)
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > partition info
> > > > for
> > > > > > topic mytopic
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0]
> > > > > > has
> > > > leader 3
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > registered for
> > > > > > topic: mytopic are 0
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with
> > > > > > no compression to [mytopic,0] [DEBUG] [main 2014-02-04
> > > > > > 17:46:01,833] Producer sending messages with correlation id 2
> > > > > > for topics [mytopic,0] to broker 3 on
> > > > 10.199.31.87:9094
> > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > at
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:1
> > > 61)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > ncPr
> > > > oducer.scala:68)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > > 1.ap
> > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > scala:101)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la:1
> > > > 01)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la:1
> > > > 01)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > ultE
> > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > at
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:1
> > > 61)
> > > > > > at
> > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > 194)
> > > > > > at
> > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > at
> > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > faul
> > > > tEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandle
> > > > r.sc
> > > > ala:72)
> > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > at
> > > > > >
> > > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > java:48)
> > > > > > at
> > > > > >
> > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer
> > > > (Kaf
> > > > kaProducerTest.java:33)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> > > > > > at
> > > > > >
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
> > > > pl.j
> > > > ava:57)
> > > > > > at
> > > > > >
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
> > > > cess
> > > > orImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > at
> > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > at
> > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.r
> > > > un(J
> > > > Unit3TestReference.java:131)
> > > > > > at
> > > > > >
> > > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > java:38)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > mote
> > > > TestRunner.java:467)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > mote
> > > > TestRunner.java:683)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteT
> > > > estR
> > > > unner.java:390)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remote
> > > > Test
> > > > Runner.java:197)
> > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > > > > request with correlation id 2 to broker 3 with data for
> > > > > > partitions [mytopic,0]
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > > Could
> > > > some
> > > > > > EC2 users here help out?
> > > > > >
> > > > > > Also, we recently added
> > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > which allows one to use a different ip for binding and
> connecting.
> > > > > > You
> > > > can
> > > > > > see if this works for you. The patch is only in trunk though.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > > > (Contingent) <
> > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > > Actually the public IP is not one of the IP configured in
> > > > > > > the Ethernet
> > > > interfaces.
> > > > > > > Only the Local IP is configured in eth0.
> > > > > > > Is there any solution to this?
> > > > > > >
> > > > > > > Ifconfig O/P:
> > > > > > >
> > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > Mask:255.255.255.192
> > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> frame:0
> > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > carrier:0
> > > > > > > collisions:0 txqueuelen:1000
> > > > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154
> > > > > > > (14.3
> > > MiB)
> > > > > > > Interrupt:25
> > > > > > >
> > > > > > > lo Link encap:Local Loopback
> > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > RX packets:1379711 errors:0 dropped:0 overruns:0
> > frame:0
> > > > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> > > carrier:0
> > > > > > > collisions:0 txqueuelen:0
> > > > > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > > > > (104.0
> > > > > > > MiB)
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Could it be a port conflict?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > Thanks for your help.
> > > > > > > > I get the following exception :
> > > > > > > > kafka.common.KafkaException: Socket server failed to bind
> > > > > > > > to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.sca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > >
> > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > >
> > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > overridden to
> > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden
> > > > > > > > to
> > > > > > > > 9092
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > overridden to
> > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > log.cleanup.interval.mins
> > > > is
> > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect
> > > > > > > > is overridden to
> > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > overridden
> > > > to 1
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > num.network.threads is overridden to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes
> > > > > > > > is overridden to
> > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log flusher every 3000 ms with the following
> > > > > > > > overrides Map()
> > > > > > > > (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > KafkaServerStable
> > > > > > > > startup. Prepare to shutdown
> > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > kafka.common.KafkaException: Socket server failed to bind
> > > > > > > > to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.sca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting
> > > > > > > > down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutting
> > > > > > > > down
> > > > > > > > (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutdown
> > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> > > > completed
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting
> > > > > > > > down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > You should should use the public IP for host.name. What's
> > > > > > > > the
> > > > error
> > > > > > > > you see during broker startup?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > I checked the faq. I did change the host.name in server
> > > > properties.
> > > > > > > > > After changing it I get ConnectException.
> > > > > > > > >
> > > > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > > > address
> > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > which is 10.x.x.x. I think because of this there is a
> > > ConnectException.
> > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > >
> > > > > > > > > What should be the IP address that is to be given in
> > > > > > host.nameproperty.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > Have you looked at
> > > > > > > > >
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > ?
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have a remote server (EC2) setup with Kafka cluster
> > setup.
> > > > > > > > > > There are
> > > > > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> > > > zookeeper
> > > > > > > > > > is running in the port 2181.
> > > > > > > > > > When I send message to the brokers from my PC, I get
> > > > > > > > > > an exception which is given below. I did a dump in the
> > > > > > > > > > remote server, the request is received in the remote
> server.
> > > > > > > > > > I am able to locally test the consumer/producer script
> > > > > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > > > > kindly help me in this
> > > > > > > > error?
> > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > >
> > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:90
> > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class
> > > > > > > > > > is overridden to kafka.serializer.StringEncoder [
> > > > > > > > > > INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> > > > overridden
> > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > > > > 16:06:50,154]
> > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id
> > > > > > > > > > 0 for 1
> > > > > > > > > > topic(s)
> > > > > > > > > > Set(mytopic)
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket
> > > > > > > > > > with SO_TIMEOUT =
> > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested
> > > > > > > > > > -1), SO_SNDBUF =
> > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > > > 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > 54.241.44.129:9093
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > > > fetched metadata for 1
> > > > > > > > > > topic(s) Set(mytopic)
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > > > > partition info for topic mytopic [DEBUG] [main
> > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072] Broker
> > > > > > > > > > partitions registered for
> > > > > > > > > > topic: mytopic are 0
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1
> > > > > > > > > > messages with no compression to [mytopic,0] [DEBUG]
> > > > > > > > > > [main
> > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > with correlation id 2 for topics [mytopic,0] to broker
> > > > > > > > > > 2 on
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > connection to
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > at
> > > > > > > > >
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > 640)
> > > > > > > > > > at
> > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scal
> > > > > > > > a:57
> > > > > > > > )
> > > > > > > > > > at
> > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > > > > nc
> > > > > > > > > Pr
> > > > > > > > > oducer.scala:68)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > scala:101)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > > > > ul
> > > > > > > > > tE
> > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > 194)
> > > > > > > > > > at
> > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scal
> > > > > > > > a:39
> > > > > > > > )
> > > > > > > > > > at
> > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > > > > fa
> > > > > > > > > ul
> > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > sc
> > > > > > > > > ala:72)
> > > > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > at
> > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > > > > at
> > > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.ja
> > > > > > > > > > va:2
> > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to
> > > > > > > > > > send producer request with correlation id 2 to broker
> > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>
Re: Reg Exception in Kafka
Posted by David Birdsong <da...@gmail.com>.
Yeah, I've only skimmed this, but I think I might have something.
All non-vpc type ec2 nodes come with an external IP address and an internal
IP address. The external IP address is what grants the node access to the
internet--makes it publicly routable. The mechanism by which the external
IP address isn't fully disclosed, but one can infer that there exists some
NAT device off the instance that maps any inbound traffic to the external
IP to the EC2 instances internal IP.
When deploying distributed systems that employ service registry(zk) in EC2,
I advise choosing between connecting only on the internal network or only
on the external network. The internal network is usually my preference for
simple privacy reasons.
..anyway, this means all *internal* name resolution needs to be figured out
by using something like /etc/hosts or using all the internal names and the
name services provide by AWS(or for the intrepid, set up internal DNS
servers.) If you go the internal route, then put all of your nodes in the
same security group.
The issue of timing out network requests smacks of a kafka node (be it a
broker or producer, I didn't parse which one) either picking up an external
IP address from zookeeper or the security groups not allowing access on the
ports needed. The default behavior of AWS security groups is to DROP which
can be confusing when you're trying to figure something out
Lastly, the mention of needing to connect to kafka(in ec2) from your office
desktop makes me think you need to step back and detangle this deployment.
Running distributed services in the cloud doesn't magically solve the
connectivity issue--especially for distributed systems that register
themselves in zookeeper.
- are you running zookeeper *and* kafka across public ip addresses?
if no, then do you have a vpn to connect your 'private' network in ec2?
- are you running zookeeper *and* kafka across private ip addresses in ec2?
if so, can they all connect with no timeouts? if so, try running a
consumer on an ec2 node.
I'll throw out a scenario that I expect you might be in:
- zk quorum in ec2, all connected to each other across private ip addresses
- kafka cluster registered into zk, broker id's list either ip addreses or
'private' names that ec2 provides
- you punched a hole zookeeper connections in your security group in ec2
- your desktop connects to zookeeper over a public ip
- assuming that works
- your desktop finds private names or ip addresses in zk for kafka brokers
- your desktop can't resolve those internal names, if it could it wouldn't
be able to route to the ip addresses
Hope this helps, good luck.
On Sun, Feb 23, 2014 at 9:16 PM, Jun Rao <ju...@gmail.com> wrote:
> Hmm, then I am not sure what happened. Anyone with EC2 experience can help?
>
> Thanks,
>
> Jun
>
>
> On Sun, Feb 23, 2014 at 6:00 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > The ports are already added in the security group. I am able to telnet
> > from the same machine I am running the producer/consume test.
> > Is there any configuration I missed?
> >
> > Thanks
> > Bala
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Saturday, February 22, 2014 12:10 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Maybe you need to add the port to the right security group?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Feb 20, 2014 at 9:58 PM, Balasubramanian Jayaraman (Contingent) <
> > balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > One point to note is that, I am trying to access the Kafka broker
> > > (located in Amazon Cloud, EC2 instance) from the Eclipse (located in
> > > my office). I am using Kafka from the trunk .
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Friday, 21 February, 2014 1:51 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > That explains why your producer hits connection timeout. Not sure why
> > > the controller to broker connection also times out though, if you can
> > > manually establish the connection.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Feb 20, 2014 at 7:37 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > All the brokers reside in the same server and are listening on the
> > > > port 10092,10093,10094. From the same machine I can connect to the
> > > > zookeeper and the brokers. But When I tried to connect from an
> > > > external machine (from Eclipse), I get an exception as communicated
> > > > earlier. I was not able to connect to any of the brokers. I get the
> > > > same exception while connecting to all the brokers.
> > > >
> > > > Regards
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Thursday, 20 February, 2014 12:05 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Can you make the connection from the controller host to the other
> > broker?
> > > > Also, what's the # open file handlers on each broker? Do you see any
> > > > "too many open file handler" error in the broker log?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I don't think so. I am able to connect to the server using a putty.
> > > > > It is VM running in Amazon cloud.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, 19 February, 2014 12:32 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Any issue with the network?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I just came from vacation.
> > > > > > When I tested it, I get a Connect Exception. The complete stack
> > > > > > trace is given below.
> > > > > >
> > > > > > Exception Stack Trace
> > > > > >
> > > > > > Java Thin Client:
> > > > > >
> > > > > > ava.net.ConnectException: Connection timed out: connect
> > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > at
> > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > at
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > )
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > nc
> > > > > Pr
> > > > > oducer.scala:68)
> > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > > > > at
> > > > > >
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > > > > at
> > > > > >
> > > > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.
> > > > > sc
> > > > > al
> > > > > a:82)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$m
> > > > > cV
> > > > > $s
> > > > > p(DefaultEventHandler.scala:67)
> > > > > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > > > > at
> > kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > > > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > > > > at
> > > > > >
> > > > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > sc
> > > > > ala:67)
> > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > at
> > > > > >
> > > > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > > java:48)
> > > > > > at
> > > > > >
> > > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer
> > > > > (K
> > > > > af
> > > > > kaProducerTest.java:33)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> > > > > > at
> > > > > >
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
> > > > > pl
> > > > > .j
> > > > > ava:57)
> > > > > > at
> > > > > >
> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
> > > > > ce
> > > > > ss
> > > > > orImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > at
> > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > at
> > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > at
> > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.r
> > > > > un
> > > > > (J
> > > > > Unit3TestReference.java:131)
> > > > > > at
> > > > > >
> > > > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > > java:38)
> > > > > > at
> > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > > mo
> > > > > te
> > > > > TestRunner.java:467)
> > > > > > at
> > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > > mo
> > > > > te
> > > > > TestRunner.java:683)
> > > > > > at
> > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteT
> > > > > es
> > > > > tR
> > > > > unner.java:390)
> > > > > > at
> > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remote
> > > > > Te
> > > > > st
> > > > > Runner.java:197)
> > > > > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata
> > > > > > with correlation id 0 for topics [Set(mytopic)] from broker
> > > > > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > > > > >
> > > > > > Kafka Server:
> > > > > > [2014-02-19 03:38:22,587] ERROR
> > > > > > [Controller-3-to-broker-3-send-thread],
> > > > > > Controller 3's connection to broker
> > > > > > id:3,host:54.xx.xx.xx,port:10094 was unsuccessful
> > > > > > (kafka.controller.RequestSendThread)
> > > > > > java.net.ConnectException: Connection timed out
> > > > > > at sun.nio.ch.Net.connect(Native Method)
> > > > > > at
> > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > > > >
> > > > > kafka.controller.RequestSendThread.connectToBroker(ControllerChann
> > > > > el
> > > > > Ma
> > > > > nager.scala:172)
> > > > > > at
> > > > > >
> > > > > kafka.controller.RequestSendThread.<init>(ControllerChannelManager
> > > > > .s
> > > > > ca
> > > > > la:115)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ControllerChannelManager.kafka$controller$Control
> > > > > le
> > > > > rC
> > > > > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ControllerChannelManager.addBroker(ControllerChan
> > > > > ne
> > > > > lM
> > > > > anager.scala:65)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > > $h
> > > > > an
> > > > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine
> > > > > .s
> > > > > ca
> > > > > la:265)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > > $h
> > > > > an
> > > > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine
> > > > > .s
> > > > > ca
> > > > > la:265)
> > > > > > at
> > scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > > $h
> > > > > an
> > > > > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > > $h
> > > > > an
> > > > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > > $h
> > > > > an
> > > > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> > > > > >
> > > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleCh
> > > > > il
> > > > > dC
> > > > > hange(ReplicaStateMachine.scala:255)
> > > > > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > > > > at
> > > > > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > > > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker
> > > > > > startup
> > > > > callback
> > > > > > for 3 (kafka.controller.KafkaController)
> > > > > > [2014-02-19 03:38:22,592] INFO [Replica state machine on
> > > > > > controller
> > > 3]:
> > > > > > Invoking state change to OnlineReplica for replicas
> > > > > > (kafka.controller.ReplicaStateMachine)
> > > > > > [2014-02-19 03:38:22,595] INFO
> > > > > > [Controller-3-to-broker-3-send-thread],
> > > > > > Starting (kafka.controller.RequestSendThread)
> > > > > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on
> > > > > > Controller
> > > 3]:
> > > > > > Broker change listener fired for path /brokers/ids with children
> > > > > > 3,2,1
> > > > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > > > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on
> > > > > > Controller
> > > 3]:
> > > > > > Newly added brokers: 2,1, deleted brokers: , all live brokers:
> > > > > > 3,2,1
> > > > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > > > > balasubramanian.jayaraman@autodesk.com]
> > > > > > Sent: Thursday, February 06, 2014 5:40 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: RE: Reg Exception in Kafka
> > > > > >
> > > > > > After correcting the spelling, it is registered to correct public
> > IP.
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > I think you mis-spelled the property name. It should be "
> > > > > > advertised.host.name", instead of advertise.host.name
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > Thanks Joel.
> > > > > > >
> > > > > > > It seems the Broker is registered to the zookeeper with the IP
> > > > > > > 10.199.31.87. The output pf the command is given below.
> > > > > > >
> > > > > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181
> > > > > > > get
> > > > > > > /brokers/ids/1
> > > > > > > Connecting to 0.0.0.0:2181
> > > > > > >
> > > > > > > WATCHER::
> > > > > > >
> > > > > > > WatchedEvent state:SyncConnected type:None path:null
> > > > > > >
> > > > > > >
> > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > > > > vers
> > > > > > > ion":1,"port":9092}
> > > > > > > cZxid = 0x2d
> > > > > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu
> > > > > > > Feb
> > > > > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength =
> > > > > > > 89
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Use zookeeper-shell script:
> > > > > > >
> > > > > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > > > > /brokers/ids/<your broker id>
> > > > > > >
> > > > > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > > > > Jayaraman
> > > > > > > (Contingent) wrote:
> > > > > > > > Where should I look for these information. From the logs, I
> > > > > > > > could see
> > > > > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka
> > > > > > > Server is started in port 9082 and bind to IP 10.x.x.x.
> > > > > > > > If I don't give the host.name in server.properties, I get "
> > > > > > > java.nio.channels.UnresolvedAddressException" and If I give
> > > > > > > the
> > > > > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > > > > It is the same behavior as in 0.8.0.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Bala
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > > > > http://10.199.31.87:9094/>)
> > > > > > > > is still the local ip, not the public one. Could you check
> > > > > > > > the broker registration in zk (
> > > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+s
> > > > > > > tr
> > > > > > > uc
> > > > > > > ture
> > > > > > > s+in+Zookeeper
> > > > > > > )
> > > > > > > > and see what's the host/port of the broker?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > I downloaded from the trunk and set up the properties
> > > > > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > > > > >
> > > > > > > > > Even after this change, I get the ConnectException. The
> > > > > > > > > details logs
> > > > > > > are
> > > > > > > > > given below. Is there any workaround for this?
> > > > > > > > >
> > > > > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > > > > 54.241.44.129:9094
> > > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully
> > > > > > > > > fetched metadata
> > > > > > > for 1
> > > > > > > > > topic(s) Set(mytopic)
> > > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > > > > partition info
> > > > > > > for
> > > > > > > > > topic mytopic
> > > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition
> > > > > > > > > [mytopic,0] has
> > > > > > > leader 3
> > > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > > > > registered for
> > > > > > > > > topic: mytopic are 0
> > > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages
> > > > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > > > 2014-02-04 17:46:01,833] Producer sending messages with
> > > > > > > > > correlation id
> > > > > > > > > 2 for topics [mytopic,0] to broker 3 on
> > > > > > > 10.199.31.87:9094
> > > > > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection
> > > > > > > > > to
> > > > > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > > > > at
> > > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:66
> > > > > > > 6)
> > > > > > > > > at
> > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > > > )
> > > > > > > > > at
> > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > at
> > > > > > > > >
> > > > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.sca
> > > > > > la
> > > > > > :1
> > > > > > 61)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSen
> > > > > > > d(
> > > > > > > Sy
> > > > > > > ncPr
> > > > > > > oducer.scala:68)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > > > $s
> > > > > > > p$
> > > > > > > 1.ap
> > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > > > $s
> > > > > > > p$
> > > > > > > 1.ap
> > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > > > $s
> > > > > > > p$
> > > > > > > 1.ap
> > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > at
> kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > scala:101)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > > .s
> > > > > > > ca
> > > > > > > la:1
> > > > > > > 01)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > > .s
> > > > > > > ca
> > > > > > > la:1
> > > > > > > 01)
> > > > > > > > > at
> kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > at
> > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$
> > > > > > > De
> > > > > > > fa
> > > > > > > ultE
> > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > > al
> > > > > > > iz
> > > > > > > edDa
> > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > > al
> > > > > > > iz
> > > > > > > edDa
> > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > scal
> > > > > > > a:80)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > scal
> > > > > > > a:80)
> > > > > > > > > at
> > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > at
> > > > > > > > >
> > > > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.sca
> > > > > > la
> > > > > > :1
> > > > > > 61)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > 194)
> > > > > > > > > at
> > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > > > )
> > > > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedDat
> > > > > > > a(
> > > > > > > De
> > > > > > > faul
> > > > > > > tEventHandler.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHa
> > > > > > > nd
> > > > > > > le
> > > > > > > r.sc
> > > > > > > ala:72)
> > > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > at
> > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > > > > java:48)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProd
> > > > > > > uc
> > > > > > > er
> > > > > > > (Kaf
> > > > > > > kaProducerTest.java:33)
> > > > > > > > > at
> > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > > Method)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess
> > > > > > > or
> > > > > > > Im
> > > > > > > pl.j
> > > > > > > ava:57)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> > > > > > > od
> > > > > > > Ac
> > > > > > > cess
> > > > > > > orImpl.java:43)
> > > > > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > > > > at
> > junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > > > > at
> > junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > > > > at
> > > > > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > > > > at
> > > > > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > > > > at
> > junit.framework.TestResult.run(TestResult.java:109)
> > > > > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > > > > at
> > > junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > > > > at
> junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReferen
> > > > > > > ce
> > > > > > > .r
> > > > > > > un(J
> > > > > > > Unit3TestReference.java:131)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > > > > java:38)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTest
> > > > > > > s(
> > > > > > > Re
> > > > > > > mote
> > > > > > > TestRunner.java:467)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTest
> > > > > > > s(
> > > > > > > Re
> > > > > > > mote
> > > > > > > TestRunner.java:683)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(Rem
> > > > > > > ot
> > > > > > > eT
> > > > > > > estR
> > > > > > > unner.java:390)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Re
> > > > > > > mo
> > > > > > > te
> > > > > > > Test
> > > > > > > Runner.java:197)
> > > > > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send
> > > > > > > > > producer request with correlation id 2 to broker 3 with
> > > > > > > > > data for partitions [mytopic,0]
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > Hmm, it's weird that EC2 only allows you to bind to local
> ip.
> > > > > > > > > Could
> > > > > > > some
> > > > > > > > > EC2 users here help out?
> > > > > > > > >
> > > > > > > > > Also, we recently added
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > > > > which allows one to use a different ip for binding and
> > > > connecting.
> > > > > > > > > You
> > > > > > > can
> > > > > > > > > see if this works for you. The patch is only in trunk
> though.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > (Contingent) <
> > > > > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > > > >
> > > > > > > > > > I don't think so. I forgot to include the ifconfig
> output.
> > > > > > > > > > Actually the public IP is not one of the IP configured
> > > > > > > > > > in the Ethernet
> > > > > > > interfaces.
> > > > > > > > > > Only the Local IP is configured in eth0.
> > > > > > > > > > Is there any solution to this?
> > > > > > > > > >
> > > > > > > > > > Ifconfig O/P:
> > > > > > > > > >
> > > > > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > > > > Mask:255.255.255.192
> > > > > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64
> > Scope:Link
> > > > > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500
> > Metric:1
> > > > > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> > > > frame:0
> > > > > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > > > > carrier:0
> > > > > > > > > > collisions:0 txqueuelen:1000
> > > > > > > > > > RX bytes:40233350 (38.3 MiB) TX
> > > > > > > > > > bytes:15089154
> > > > > > > > > > (14.3
> > > > > > MiB)
> > > > > > > > > > Interrupt:25
> > > > > > > > > >
> > > > > > > > > > lo Link encap:Local Loopback
> > > > > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > > > > RX packets:1379711 errors:0 dropped:0
> > > > > > > > > > overruns:0
> > > > > frame:0
> > > > > > > > > > TX packets:1379711 errors:0 dropped:0
> > > > > > > > > > overruns:0
> > > > > > carrier:0
> > > > > > > > > > collisions:0 txqueuelen:0
> > > > > > > > > > RX bytes:109133672 (104.0 MiB) TX
> > > > > > > > > > bytes:109133672
> > > > > > > > > > (104.0
> > > > > > > > > > MiB)
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > > >
> > > > > > > > > > Could it be a port conflict?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian
> > > > > > > > > > Jayaraman
> > > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Jun,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for your help.
> > > > > > > > > > > I get the following exception :
> > > > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > > > bind to
> > > > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > > > at
> > > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > > la
> > > > > > > > > > :1
> > > > > > > > > > 88)
> > > > > > > > > > > at
> > > > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > > > at
> > > > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > > > at
> > > > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable
> > > > > > > .s
> > > > > > > ca
> > > > > > > la:3
> > > > > > > 4)
> > > > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > > > ja
> > > > > > > va:1
> > > > > > > 39)
> > > > > > > > > > > at
> > > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > > ja
> > > > > > > > > > va
> > > > > > > > > > :77)
> > > > > > > > > > > at
> > > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > > ja
> > > > > > > > > > va
> > > > > > > > > > :70)
> > > > > > > > > > > at
> > > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > > la
> > > > > > > > > > :1
> > > > > > > > > > 84)
> > > > > > > > > > > ... 6 more
> > > > > > > > > > >
> > > > > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > > > > >
> > > > > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > > > > overridden to
> > > > > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is
> > > > > > > > > > > overridden to
> > > > > > > > > > > 9092
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads
> > > > > > > > > > > is
> > > > > > > overridden
> > > > > > > > > > > to 2
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > > > > overridden to
> > > > > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > > > log.cleanup.interval.mins
> > > > > > > is
> > > > > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > > > zookeeper.connect is overridden to
> > > > > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > > > > overridden
> > > > > > > to 1
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > > > num.network.threads is overridden to 2
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > > > zookeeper.connection.timeout.msis overridden to
> > > > > > > > > > > 1000000
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions
> > > > > > > > > > > is
> > > > > > > overridden
> > > > > > > > > > > to 2
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,182] INFO Property
> > > > > > > > > > > log.segment.bytes is overridden to
> > > > > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1],
> > > > > > > > > > > Starting
> > > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker
> > > > > > > > > > > 1] Starting log cleaner every 60000 ms
> > > > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker
> > > > > > > > > > > 1] Starting log flusher every 3000 ms with the
> > > > > > > > > > > following overrides Map()
> > > > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > > > > KafkaServerStable
> > > > > > > > > > > startup. Prepare to shutdown
> > > > > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > > > bind to
> > > > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > > > at
> > > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > > la
> > > > > > > > > > :1
> > > > > > > > > > 88)
> > > > > > > > > > > at
> > > > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > > > at
> > > > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > > > at
> > > > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable
> > > > > > > .s
> > > > > > > ca
> > > > > > > la:3
> > > > > > > 4)
> > > > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > > > ja
> > > > > > > va:1
> > > > > > > 39)
> > > > > > > > > > > at
> > > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > > ja
> > > > > > > > > > va
> > > > > > > > > > :77)
> > > > > > > > > > > at
> > > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > > ja
> > > > > > > > > > va
> > > > > > > > > > :70)
> > > > > > > > > > > at
> > > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > > la
> > > > > > > > > > :1
> > > > > > > > > > 84)
> > > > > > > > > > > ... 6 more
> > > > > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1],
> > > > > > > > > > > Shutting down
> > > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on
> > > > > > > > > > > Broker 1],
> > > > > > > Shutting
> > > > > > > > > > > down
> > > > > > > > > > > (kafka.network.SocketServer)
> > > > > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on
> > > > > > > > > > > Broker 1],
> > > > > > > Shutdown
> > > > > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka
> > > > > > > > > > > scheduler
> > > > > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut
> > > > > > > > > > > down
> > > > > > > completed
> > > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1],
> > > > > > > > > > > Shutting down
> > > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > > >
> > > > > > > > > > > Regards
> > > > > > > > > > > Bala
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > > > >
> > > > > > > > > > > You should should use the public IP for host.name.
> > > > > > > > > > > What's the
> > > > > > > error
> > > > > > > > > > > you see during broker startup?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian
> > > > > > > > > > > Jayaraman
> > > > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > I checked the faq. I did change the host.name in
> > > > > > > > > > > > server
> > > > > > > properties.
> > > > > > > > > > > > After changing it I get ConnectException.
> > > > > > > > > > > >
> > > > > > > > > > > > The problem here is in EC2 have a different public
> > > > > > > > > > > > IP address
> > > > > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > > > > which is 10.x.x.x. I think because of this there is
> > > > > > > > > > > > a
> > > > > > ConnectException.
> > > > > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > > > > >
> > > > > > > > > > > > What should be the IP address that is to be given in
> > > > > > > > > host.nameproperty.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Bala
> > > > > > > > > > > >
> > > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > > > > >
> > > > > > > > > > > > Have you looked at
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC
> > > > > > > 2,
> > > > > > > wh
> > > > > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > > > > ?
> > > > > > > > > > > >
> > > > > > > > > > > > thanks,
> > > > > > > > > > > >
> > > > > > > > > > > > Jun
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > > > > Jayaraman
> > > > > > > > > > > > (Contingent) <
> > > > > > > > > > > > balasubramanian.jayaraman@autodesk.com>
> > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have a remote server (EC2) setup with Kafka
> > > > > > > > > > > > > cluster
> > > > > setup.
> > > > > > > > > > > > > There are
> > > > > > > > > > > > > 3 brokers each running in the port 9092,9093,9094.
> > > > > > > > > > > > > The
> > > > > > > zookeeper
> > > > > > > > > > > > > is running in the port 2181.
> > > > > > > > > > > > > When I send message to the brokers from my PC, I
> > > > > > > > > > > > > get an exception which is given below. I did a
> > > > > > > > > > > > > dump in the remote server, the request is received
> > > > > > > > > > > > > in the remote
> > > > server.
> > > > > > > > > > > > > I am able to locally test the consumer/producer
> > > > > > > > > > > > > script present in the bin folder. What am I
> missing?
> > > > > > > > > > > > > Can you kindly help me in this
> > > > > > > > > > > error?
> > > > > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,
> 54.241.44.129:
> > > > > > > > > > > > > 90
> > > > > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > > > request.required.acks is overridden to 1 [ INFO]
> > > > > > > > > > > > > [main
> > > > > > > > > > > > > 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > > > key.serializer.class is overridden to
> > > > > > > > > > > > > kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class
> > > > > > > > > > > > > is
> > > > > > > overridden
> > > > > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main
> > > > > > > > > > > > > 2014-01-27 16:06:50,154]
> > > > > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main
> > > > > > > > > > > > > 2014-01-27 16:06:50,298] Handling 1 events [ INFO]
> > > > > > > > > > > > > [main
> > > > > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation
> > > > > > > > > > > > > id
> > > > > > > > > > > > > 0 for 1
> > > > > > > > > > > > > topic(s)
> > > > > > > > > > > > > Set(mytopic)
> > > > > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created
> > > > > > > > > > > > > socket with SO_TIMEOUT =
> > > > > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192
> > > > > > > > > > > > > (requested -1), SO_SNDBUF =
> > > > > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected
> > > > > > > > > > > > > to 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > > > > 54.241.44.129:9093 [DEBUG] [main 2014-01-27
> > > > > > > > > > > > > 15:59:44,025] Successfully fetched metadata for 1
> > > > > > > > > > > > > topic(s) Set(mytopic) [DEBUG] [main 2014-01-27
> > > > > > > > > > > > > 15:59:44,058] Getting broker partition info for
> > > > > > > > > > > > > topic mytopic [DEBUG] [main
> > > > > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072]
> > > > > > > > > > > > > Broker partitions registered for
> > > > > > > > > > > > > topic: mytopic are 0 [DEBUG] [main 2014-01-27
> > > > > > > > > > > > > 15:59:44,091] Sending 1 messages with no
> > > > > > > > > > > > > compression to [mytopic,0] [DEBUG] [main
> > > > > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > > > > with correlation id 2 for topics [mytopic,0] to
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > 2 on
> > > > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > > > > connection to
> > > > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > > > unsuccessful
> > > java.nio.channels.UnresolvedAddressException
> > > > > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > > > > 640)
> > > > > > > > > > > > > at
> > > > > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.
> > > > > > > > > > > sc
> > > > > > > > > > > al
> > > > > > > > > > > a:57
> > > > > > > > > > > )
> > > > > > > > > > > > > at
> > > > > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146
> > > > > > > > > )
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.s
> > > > > > > ca
> > > > > > > la
> > > > > > > :161
> > > > > > > > > > )
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSen
> > > > > > > d(
> > > > > > > Sy
> > > > > > > > > > > > nc
> > > > > > > > > > > > Pr
> > > > > > > > > > > > oducer.scala:68)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > > > ap
> > > > > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > > > ap
> > > > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > > > ap
> > > > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > > > at
> > > > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > > > > scala:101)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > > .s
> > > > > > > ca
> > > > > > > la
> > > > > > > > > > > :1
> > > > > > > > > > > 01)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > > .s
> > > > > > > ca
> > > > > > > la
> > > > > > > > > > > :1
> > > > > > > > > > > 01)
> > > > > > > > > > > > > at
> > > > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > > > at
> > > > > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$
> > > > > > > De
> > > > > > > fa
> > > > > > > > > > > > ul
> > > > > > > > > > > > tE
> > > > > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > > al
> > > > > > > iz
> > > > > > > > > > > > ed
> > > > > > > > > > > > Da
> > > > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > > al
> > > > > > > iz
> > > > > > > > > > > > ed
> > > > > > > > > > > > Da
> > > > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > > > sc
> > > > > > > > > > > > al
> > > > > > > > > > > > a:80)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > > > sc
> > > > > > > > > > > > al
> > > > > > > > > > > > a:80)
> > > > > > > > > > > > > at
> > > > > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631
> > > > > > > > > )
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.s
> > > > > > > ca
> > > > > > > la
> > > > > > > :161
> > > > > > > > > > )
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > > > > 194)
> > > > > > > > > > > > > at
> > > > > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.
> > > > > > > > > > > sc
> > > > > > > > > > > al
> > > > > > > > > > > a:39
> > > > > > > > > > > )
> > > > > > > > > > > > > at
> > > > > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedDat
> > > > > > > a(
> > > > > > > De
> > > > > > > > > > > > fa
> > > > > > > > > > > > ul
> > > > > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > > > > sc
> > > > > > > > > > > > ala:72)
> > > > > > > > > > > > > at
> > > kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > > > > at
> > > > > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> > > > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java
> > > > > > > :3
> > > > > > > 9)
> > > > > > > > > > > > > at
> > > > > > > > > > > > >
> kafka.test.KafkaProducerTest.main(KafkaProducerTest.
> > > > > > > > > > > > > ja
> > > > > > > > > > > > > va:2
> > > > > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed
> > > > > > > > > > > > > to send producer request with correlation id 2 to
> > > > > > > > > > > > > broker
> > > > > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > Bala
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Reg Exception in Kafka
Posted by Jun Rao <ju...@gmail.com>.
Hmm, then I am not sure what happened. Anyone with EC2 experience can help?
Thanks,
Jun
On Sun, Feb 23, 2014 at 6:00 PM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> The ports are already added in the security group. I am able to telnet
> from the same machine I am running the producer/consume test.
> Is there any configuration I missed?
>
> Thanks
> Bala
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Saturday, February 22, 2014 12:10 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Maybe you need to add the port to the right security group?
>
> Thanks,
>
> Jun
>
>
> On Thu, Feb 20, 2014 at 9:58 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > One point to note is that, I am trying to access the Kafka broker
> > (located in Amazon Cloud, EC2 instance) from the Eclipse (located in
> > my office). I am using Kafka from the trunk .
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Friday, 21 February, 2014 1:51 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > That explains why your producer hits connection timeout. Not sure why
> > the controller to broker connection also times out though, if you can
> > manually establish the connection.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, Feb 20, 2014 at 7:37 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > All the brokers reside in the same server and are listening on the
> > > port 10092,10093,10094. From the same machine I can connect to the
> > > zookeeper and the brokers. But When I tried to connect from an
> > > external machine (from Eclipse), I get an exception as communicated
> > > earlier. I was not able to connect to any of the brokers. I get the
> > > same exception while connecting to all the brokers.
> > >
> > > Regards
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, 20 February, 2014 12:05 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Can you make the connection from the controller host to the other
> broker?
> > > Also, what's the # open file handlers on each broker? Do you see any
> > > "too many open file handler" error in the broker log?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I don't think so. I am able to connect to the server using a putty.
> > > > It is VM running in Amazon cloud.
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, 19 February, 2014 12:32 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Any issue with the network?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I just came from vacation.
> > > > > When I tested it, I get a Connect Exception. The complete stack
> > > > > trace is given below.
> > > > >
> > > > > Exception Stack Trace
> > > > >
> > > > > Java Thin Client:
> > > > >
> > > > > ava.net.ConnectException: Connection timed out: connect
> > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > at
> > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > at
> > > > >
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > )
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > nc
> > > > Pr
> > > > oducer.scala:68)
> > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > > > at
> > > > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > > > at
> > > > >
> > > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.
> > > > sc
> > > > al
> > > > a:82)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$m
> > > > cV
> > > > $s
> > > > p(DefaultEventHandler.scala:67)
> > > > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > > > at
> kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > sc
> > > > ala:67)
> > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > at
> > > > >
> > > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > java:48)
> > > > > at
> > > > >
> > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer
> > > > (K
> > > > af
> > > > kaProducerTest.java:33)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > > > > at
> > > > >
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
> > > > pl
> > > > .j
> > > > ava:57)
> > > > > at
> > > > >
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
> > > > ce
> > > > ss
> > > > orImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > at
> junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > at
> > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > at
> > > > >
> > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.r
> > > > un
> > > > (J
> > > > Unit3TestReference.java:131)
> > > > > at
> > > > >
> > > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > java:38)
> > > > > at
> > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > mo
> > > > te
> > > > TestRunner.java:467)
> > > > > at
> > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > mo
> > > > te
> > > > TestRunner.java:683)
> > > > > at
> > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteT
> > > > es
> > > > tR
> > > > unner.java:390)
> > > > > at
> > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remote
> > > > Te
> > > > st
> > > > Runner.java:197)
> > > > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata
> > > > > with correlation id 0 for topics [Set(mytopic)] from broker
> > > > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > > > >
> > > > > Kafka Server:
> > > > > [2014-02-19 03:38:22,587] ERROR
> > > > > [Controller-3-to-broker-3-send-thread],
> > > > > Controller 3's connection to broker
> > > > > id:3,host:54.xx.xx.xx,port:10094 was unsuccessful
> > > > > (kafka.controller.RequestSendThread)
> > > > > java.net.ConnectException: Connection timed out
> > > > > at sun.nio.ch.Net.connect(Native Method)
> > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > at
> > > > >
> > > > kafka.controller.RequestSendThread.connectToBroker(ControllerChann
> > > > el
> > > > Ma
> > > > nager.scala:172)
> > > > > at
> > > > >
> > > > kafka.controller.RequestSendThread.<init>(ControllerChannelManager
> > > > .s
> > > > ca
> > > > la:115)
> > > > > at
> > > > >
> > > > kafka.controller.ControllerChannelManager.kafka$controller$Control
> > > > le
> > > > rC
> > > > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > > > at
> > > > >
> > > > kafka.controller.ControllerChannelManager.addBroker(ControllerChan
> > > > ne
> > > > lM
> > > > anager.scala:65)
> > > > > at
> > > > >
> > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > $h
> > > > an
> > > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine
> > > > .s
> > > > ca
> > > > la:265)
> > > > > at
> > > > >
> > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > $h
> > > > an
> > > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine
> > > > .s
> > > > ca
> > > > la:265)
> > > > > at
> scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > > > at
> > > > >
> > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > $h
> > > > an
> > > > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > > > at
> > > > >
> > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > $h
> > > > an
> > > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > > at
> > > > >
> > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > > $h
> > > > an
> > > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at
> > > > >
> > > > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleCh
> > > > il
> > > > dC
> > > > hange(ReplicaStateMachine.scala:255)
> > > > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > > > at
> > > > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker
> > > > > startup
> > > > callback
> > > > > for 3 (kafka.controller.KafkaController)
> > > > > [2014-02-19 03:38:22,592] INFO [Replica state machine on
> > > > > controller
> > 3]:
> > > > > Invoking state change to OnlineReplica for replicas
> > > > > (kafka.controller.ReplicaStateMachine)
> > > > > [2014-02-19 03:38:22,595] INFO
> > > > > [Controller-3-to-broker-3-send-thread],
> > > > > Starting (kafka.controller.RequestSendThread)
> > > > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on
> > > > > Controller
> > 3]:
> > > > > Broker change listener fired for path /brokers/ids with children
> > > > > 3,2,1
> > > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on
> > > > > Controller
> > 3]:
> > > > > Newly added brokers: 2,1, deleted brokers: , all live brokers:
> > > > > 3,2,1
> > > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > > > balasubramanian.jayaraman@autodesk.com]
> > > > > Sent: Thursday, February 06, 2014 5:40 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: RE: Reg Exception in Kafka
> > > > >
> > > > > After correcting the spelling, it is registered to correct public
> IP.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > I think you mis-spelled the property name. It should be "
> > > > > advertised.host.name", instead of advertise.host.name
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > Thanks Joel.
> > > > > >
> > > > > > It seems the Broker is registered to the zookeeper with the IP
> > > > > > 10.199.31.87. The output pf the command is given below.
> > > > > >
> > > > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181
> > > > > > get
> > > > > > /brokers/ids/1
> > > > > > Connecting to 0.0.0.0:2181
> > > > > >
> > > > > > WATCHER::
> > > > > >
> > > > > > WatchedEvent state:SyncConnected type:None path:null
> > > > > >
> > > > > >
> {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > > > vers
> > > > > > ion":1,"port":9092}
> > > > > > cZxid = 0x2d
> > > > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu
> > > > > > Feb
> > > > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength =
> > > > > > 89
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Use zookeeper-shell script:
> > > > > >
> > > > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > > > /brokers/ids/<your broker id>
> > > > > >
> > > > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > > > Jayaraman
> > > > > > (Contingent) wrote:
> > > > > > > Where should I look for these information. From the logs, I
> > > > > > > could see
> > > > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka
> > > > > > Server is started in port 9082 and bind to IP 10.x.x.x.
> > > > > > > If I don't give the host.name in server.properties, I get "
> > > > > > java.nio.channels.UnresolvedAddressException" and If I give
> > > > > > the
> > > > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > > > It is the same behavior as in 0.8.0.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > > > http://10.199.31.87:9094/>)
> > > > > > > is still the local ip, not the public one. Could you check
> > > > > > > the broker registration in zk (
> > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+s
> > > > > > tr
> > > > > > uc
> > > > > > ture
> > > > > > s+in+Zookeeper
> > > > > > )
> > > > > > > and see what's the host/port of the broker?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > I downloaded from the trunk and set up the properties
> > > > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > > > >
> > > > > > > > Even after this change, I get the ConnectException. The
> > > > > > > > details logs
> > > > > > are
> > > > > > > > given below. Is there any workaround for this?
> > > > > > > >
> > > > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > > > 54.241.44.129:9094
> > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully
> > > > > > > > fetched metadata
> > > > > > for 1
> > > > > > > > topic(s) Set(mytopic)
> > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > > > partition info
> > > > > > for
> > > > > > > > topic mytopic
> > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition
> > > > > > > > [mytopic,0] has
> > > > > > leader 3
> > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > > > registered for
> > > > > > > > topic: mytopic are 0
> > > > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages
> > > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > > 2014-02-04 17:46:01,833] Producer sending messages with
> > > > > > > > correlation id
> > > > > > > > 2 for topics [mytopic,0] to broker 3 on
> > > > > > 10.199.31.87:9094
> > > > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection
> > > > > > > > to
> > > > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > > > at
> > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:66
> > > > > > 6)
> > > > > > > > at
> > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > > )
> > > > > > > > at
> > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > at
> > > > > > > >
> > > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.sca
> > > > > la
> > > > > :1
> > > > > 61)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSen
> > > > > > d(
> > > > > > Sy
> > > > > > ncPr
> > > > > > oducer.scala:68)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > > $s
> > > > > > p$
> > > > > > 1.ap
> > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > > $s
> > > > > > p$
> > > > > > 1.ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > > $s
> > > > > > p$
> > > > > > 1.ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > scala:101)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > .s
> > > > > > ca
> > > > > > la:1
> > > > > > 01)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > .s
> > > > > > ca
> > > > > > la:1
> > > > > > 01)
> > > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > at
> > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$
> > > > > > De
> > > > > > fa
> > > > > > ultE
> > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > al
> > > > > > iz
> > > > > > edDa
> > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > al
> > > > > > iz
> > > > > > edDa
> > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > scal
> > > > > > a:80)
> > > > > > > > at
> > > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > scal
> > > > > > a:80)
> > > > > > > > at
> > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > at
> > > > > > > >
> > > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.sca
> > > > > la
> > > > > :1
> > > > > 61)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > 194)
> > > > > > > > at
> > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > > )
> > > > > > > > at
> > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedDat
> > > > > > a(
> > > > > > De
> > > > > > faul
> > > > > > tEventHandler.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHa
> > > > > > nd
> > > > > > le
> > > > > > r.sc
> > > > > > ala:72)
> > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > at
> > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > > > java:48)
> > > > > > > > at
> > > > > > > >
> > > > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProd
> > > > > > uc
> > > > > > er
> > > > > > (Kaf
> > > > > > kaProducerTest.java:33)
> > > > > > > > at
> > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > > Method)
> > > > > > > > at
> > > > > > > >
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess
> > > > > > or
> > > > > > Im
> > > > > > pl.j
> > > > > > ava:57)
> > > > > > > > at
> > > > > > > >
> > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> > > > > > od
> > > > > > Ac
> > > > > > cess
> > > > > > orImpl.java:43)
> > > > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > > > at
> junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > > > at
> junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > > > at
> > > > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > > > at
> > > > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > > > at
> junit.framework.TestResult.run(TestResult.java:109)
> > > > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > > > at
> > junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > > > at
> > > > > > > >
> > > > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReferen
> > > > > > ce
> > > > > > .r
> > > > > > un(J
> > > > > > Unit3TestReference.java:131)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > > > java:38)
> > > > > > > > at
> > > > > > > >
> > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTest
> > > > > > s(
> > > > > > Re
> > > > > > mote
> > > > > > TestRunner.java:467)
> > > > > > > > at
> > > > > > > >
> > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTest
> > > > > > s(
> > > > > > Re
> > > > > > mote
> > > > > > TestRunner.java:683)
> > > > > > > > at
> > > > > > > >
> > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(Rem
> > > > > > ot
> > > > > > eT
> > > > > > estR
> > > > > > unner.java:390)
> > > > > > > > at
> > > > > > > >
> > > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Re
> > > > > > mo
> > > > > > te
> > > > > > Test
> > > > > > Runner.java:197)
> > > > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send
> > > > > > > > producer request with correlation id 2 to broker 3 with
> > > > > > > > data for partitions [mytopic,0]
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > > > > Could
> > > > > > some
> > > > > > > > EC2 users here help out?
> > > > > > > >
> > > > > > > > Also, we recently added
> > > > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > > > which allows one to use a different ip for binding and
> > > connecting.
> > > > > > > > You
> > > > > > can
> > > > > > > > see if this works for you. The patch is only in trunk though.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian
> > > > > > > > Jayaraman
> > > > > > (Contingent) <
> > > > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > > >
> > > > > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > > > > Actually the public IP is not one of the IP configured
> > > > > > > > > in the Ethernet
> > > > > > interfaces.
> > > > > > > > > Only the Local IP is configured in eth0.
> > > > > > > > > Is there any solution to this?
> > > > > > > > >
> > > > > > > > > Ifconfig O/P:
> > > > > > > > >
> > > > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > > > Mask:255.255.255.192
> > > > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64
> Scope:Link
> > > > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500
> Metric:1
> > > > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> > > frame:0
> > > > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > > > carrier:0
> > > > > > > > > collisions:0 txqueuelen:1000
> > > > > > > > > RX bytes:40233350 (38.3 MiB) TX
> > > > > > > > > bytes:15089154
> > > > > > > > > (14.3
> > > > > MiB)
> > > > > > > > > Interrupt:25
> > > > > > > > >
> > > > > > > > > lo Link encap:Local Loopback
> > > > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > > > RX packets:1379711 errors:0 dropped:0
> > > > > > > > > overruns:0
> > > > frame:0
> > > > > > > > > TX packets:1379711 errors:0 dropped:0
> > > > > > > > > overruns:0
> > > > > carrier:0
> > > > > > > > > collisions:0 txqueuelen:0
> > > > > > > > > RX bytes:109133672 (104.0 MiB) TX
> > > > > > > > > bytes:109133672
> > > > > > > > > (104.0
> > > > > > > > > MiB)
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > Could it be a port conflict?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > Jun,
> > > > > > > > > >
> > > > > > > > > > Thanks for your help.
> > > > > > > > > > I get the following exception :
> > > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > > bind to
> > > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > > at
> > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > la
> > > > > > > > > :1
> > > > > > > > > 88)
> > > > > > > > > > at
> > > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > > at
> > > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > > at
> > > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable
> > > > > > .s
> > > > > > ca
> > > > > > la:3
> > > > > > 4)
> > > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > > ja
> > > > > > va:1
> > > > > > 39)
> > > > > > > > > > at
> > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > ja
> > > > > > > > > va
> > > > > > > > > :77)
> > > > > > > > > > at
> > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > ja
> > > > > > > > > va
> > > > > > > > > :70)
> > > > > > > > > > at
> > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > la
> > > > > > > > > :1
> > > > > > > > > 84)
> > > > > > > > > > ... 6 more
> > > > > > > > > >
> > > > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > > > >
> > > > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > > > overridden to
> > > > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is
> > > > > > > > > > overridden to
> > > > > > > > > > 9092
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads
> > > > > > > > > > is
> > > > > > overridden
> > > > > > > > > > to 2
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > > > overridden to
> > > > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > > log.cleanup.interval.mins
> > > > > > is
> > > > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > > zookeeper.connect is overridden to
> > > > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > > > overridden
> > > > > > to 1
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > > num.network.threads is overridden to 2
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > > zookeeper.connection.timeout.msis overridden to
> > > > > > > > > > 1000000
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions
> > > > > > > > > > is
> > > > > > overridden
> > > > > > > > > > to 2
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,182] INFO Property
> > > > > > > > > > log.segment.bytes is overridden to
> > > > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1],
> > > > > > > > > > Starting
> > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker
> > > > > > > > > > 1] Starting log cleaner every 60000 ms
> > > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker
> > > > > > > > > > 1] Starting log flusher every 3000 ms with the
> > > > > > > > > > following overrides Map()
> > > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > > > KafkaServerStable
> > > > > > > > > > startup. Prepare to shutdown
> > > > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > > bind to
> > > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > > at
> > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > la
> > > > > > > > > :1
> > > > > > > > > 88)
> > > > > > > > > > at
> > > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > > at
> > > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > > at
> > > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable
> > > > > > .s
> > > > > > ca
> > > > > > la:3
> > > > > > 4)
> > > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > > ja
> > > > > > va:1
> > > > > > 39)
> > > > > > > > > > at
> > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > ja
> > > > > > > > > va
> > > > > > > > > :77)
> > > > > > > > > > at
> > > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > > ja
> > > > > > > > > va
> > > > > > > > > :70)
> > > > > > > > > > at
> > > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > > la
> > > > > > > > > :1
> > > > > > > > > 84)
> > > > > > > > > > ... 6 more
> > > > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1],
> > > > > > > > > > Shutting down
> > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on
> > > > > > > > > > Broker 1],
> > > > > > Shutting
> > > > > > > > > > down
> > > > > > > > > > (kafka.network.SocketServer)
> > > > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on
> > > > > > > > > > Broker 1],
> > > > > > Shutdown
> > > > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka
> > > > > > > > > > scheduler
> > > > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut
> > > > > > > > > > down
> > > > > > completed
> > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1],
> > > > > > > > > > Shutting down
> > > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > >
> > > > > > > > > > Regards
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > > >
> > > > > > > > > > You should should use the public IP for host.name.
> > > > > > > > > > What's the
> > > > > > error
> > > > > > > > > > you see during broker startup?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian
> > > > > > > > > > Jayaraman
> > > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > > wrote:
> > > > > > > > > >
> > > > > > > > > > > I checked the faq. I did change the host.name in
> > > > > > > > > > > server
> > > > > > properties.
> > > > > > > > > > > After changing it I get ConnectException.
> > > > > > > > > > >
> > > > > > > > > > > The problem here is in EC2 have a different public
> > > > > > > > > > > IP address
> > > > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > > > which is 10.x.x.x. I think because of this there is
> > > > > > > > > > > a
> > > > > ConnectException.
> > > > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > > > >
> > > > > > > > > > > What should be the IP address that is to be given in
> > > > > > > > host.nameproperty.
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Bala
> > > > > > > > > > >
> > > > > > > > > > > -----Original Message-----
> > > > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > > > >
> > > > > > > > > > > Have you looked at
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC
> > > > > > 2,
> > > > > > wh
> > > > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > > > ?
> > > > > > > > > > >
> > > > > > > > > > > thanks,
> > > > > > > > > > >
> > > > > > > > > > > Jun
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > > > Jayaraman
> > > > > > > > > > > (Contingent) <
> > > > > > > > > > > balasubramanian.jayaraman@autodesk.com>
> > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > I have a remote server (EC2) setup with Kafka
> > > > > > > > > > > > cluster
> > > > setup.
> > > > > > > > > > > > There are
> > > > > > > > > > > > 3 brokers each running in the port 9092,9093,9094.
> > > > > > > > > > > > The
> > > > > > zookeeper
> > > > > > > > > > > > is running in the port 2181.
> > > > > > > > > > > > When I send message to the brokers from my PC, I
> > > > > > > > > > > > get an exception which is given below. I did a
> > > > > > > > > > > > dump in the remote server, the request is received
> > > > > > > > > > > > in the remote
> > > server.
> > > > > > > > > > > > I am able to locally test the consumer/producer
> > > > > > > > > > > > script present in the bin folder. What am I missing?
> > > > > > > > > > > > Can you kindly help me in this
> > > > > > > > > > error?
> > > > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > > > >
> > > > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:
> > > > > > > > > > > > 90
> > > > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > > request.required.acks is overridden to 1 [ INFO]
> > > > > > > > > > > > [main
> > > > > > > > > > > > 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > > key.serializer.class is overridden to
> > > > > > > > > > > > kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class
> > > > > > > > > > > > is
> > > > > > overridden
> > > > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main
> > > > > > > > > > > > 2014-01-27 16:06:50,154]
> > > > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main
> > > > > > > > > > > > 2014-01-27 16:06:50,298] Handling 1 events [ INFO]
> > > > > > > > > > > > [main
> > > > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from
> > > > > > > > > > > > broker
> > > > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation
> > > > > > > > > > > > id
> > > > > > > > > > > > 0 for 1
> > > > > > > > > > > > topic(s)
> > > > > > > > > > > > Set(mytopic)
> > > > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created
> > > > > > > > > > > > socket with SO_TIMEOUT =
> > > > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192
> > > > > > > > > > > > (requested -1), SO_SNDBUF =
> > > > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected
> > > > > > > > > > > > to 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > > > 54.241.44.129:9093 [DEBUG] [main 2014-01-27
> > > > > > > > > > > > 15:59:44,025] Successfully fetched metadata for 1
> > > > > > > > > > > > topic(s) Set(mytopic) [DEBUG] [main 2014-01-27
> > > > > > > > > > > > 15:59:44,058] Getting broker partition info for
> > > > > > > > > > > > topic mytopic [DEBUG] [main
> > > > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072]
> > > > > > > > > > > > Broker partitions registered for
> > > > > > > > > > > > topic: mytopic are 0 [DEBUG] [main 2014-01-27
> > > > > > > > > > > > 15:59:44,091] Sending 1 messages with no
> > > > > > > > > > > > compression to [mytopic,0] [DEBUG] [main
> > > > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > > > with correlation id 2 for topics [mytopic,0] to
> > > > > > > > > > > > broker
> > > > > > > > > > > > 2 on
> > > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > > > connection to
> > > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > > unsuccessful
> > java.nio.channels.UnresolvedAddressException
> > > > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > > > at
> > > > > > > > > > >
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > > > 640)
> > > > > > > > > > > > at
> > > > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.
> > > > > > > > > > sc
> > > > > > > > > > al
> > > > > > > > > > a:57
> > > > > > > > > > )
> > > > > > > > > > > > at
> > > > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146
> > > > > > > > )
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.s
> > > > > > ca
> > > > > > la
> > > > > > :161
> > > > > > > > > )
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSen
> > > > > > d(
> > > > > > Sy
> > > > > > > > > > > nc
> > > > > > > > > > > Pr
> > > > > > > > > > > oducer.scala:68)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > > ap
> > > > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > > ap
> > > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > > ap
> > > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > > at
> > > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > > > scala:101)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > .s
> > > > > > ca
> > > > > > la
> > > > > > > > > > :1
> > > > > > > > > > 01)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > > .s
> > > > > > ca
> > > > > > la
> > > > > > > > > > :1
> > > > > > > > > > 01)
> > > > > > > > > > > > at
> > > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > > at
> > > > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$
> > > > > > De
> > > > > > fa
> > > > > > > > > > > ul
> > > > > > > > > > > tE
> > > > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > al
> > > > > > iz
> > > > > > > > > > > ed
> > > > > > > > > > > Da
> > > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > > al
> > > > > > iz
> > > > > > > > > > > ed
> > > > > > > > > > > Da
> > > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > > sc
> > > > > > > > > > > al
> > > > > > > > > > > a:80)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > > sc
> > > > > > > > > > > al
> > > > > > > > > > > a:80)
> > > > > > > > > > > > at
> > > > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631
> > > > > > > > )
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > >
> > > > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.s
> > > > > > ca
> > > > > > la
> > > > > > :161
> > > > > > > > > )
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > >
> > > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > > > 194)
> > > > > > > > > > > > at
> > > > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.
> > > > > > > > > > sc
> > > > > > > > > > al
> > > > > > > > > > a:39
> > > > > > > > > > )
> > > > > > > > > > > > at
> > > > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedDat
> > > > > > a(
> > > > > > De
> > > > > > > > > > > fa
> > > > > > > > > > > ul
> > > > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > > > sc
> > > > > > > > > > > ala:72)
> > > > > > > > > > > > at
> > kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > > > at
> > > > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > > > at
> > > > > > > > > > > >
> > > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java
> > > > > > :3
> > > > > > 9)
> > > > > > > > > > > > at
> > > > > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.
> > > > > > > > > > > > ja
> > > > > > > > > > > > va:2
> > > > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed
> > > > > > > > > > > > to send producer request with correlation id 2 to
> > > > > > > > > > > > broker
> > > > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks
> > > > > > > > > > > > Bala
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
The ports are already added in the security group. I am able to telnet from the same machine I am running the producer/consume test.
Is there any configuration I missed?
Thanks
Bala
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Saturday, February 22, 2014 12:10 AM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
Maybe you need to add the port to the right security group?
Thanks,
Jun
On Thu, Feb 20, 2014 at 9:58 PM, Balasubramanian Jayaraman (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> One point to note is that, I am trying to access the Kafka broker
> (located in Amazon Cloud, EC2 instance) from the Eclipse (located in
> my office). I am using Kafka from the trunk .
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, 21 February, 2014 1:51 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> That explains why your producer hits connection timeout. Not sure why
> the controller to broker connection also times out though, if you can
> manually establish the connection.
>
> Thanks,
>
> Jun
>
>
> On Thu, Feb 20, 2014 at 7:37 PM, Balasubramanian Jayaraman
> (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
>
> > All the brokers reside in the same server and are listening on the
> > port 10092,10093,10094. From the same machine I can connect to the
> > zookeeper and the brokers. But When I tried to connect from an
> > external machine (from Eclipse), I get an exception as communicated
> > earlier. I was not able to connect to any of the brokers. I get the
> > same exception while connecting to all the brokers.
> >
> > Regards
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, 20 February, 2014 12:05 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Can you make the connection from the controller host to the other broker?
> > Also, what's the # open file handlers on each broker? Do you see any
> > "too many open file handler" error in the broker log?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I don't think so. I am able to connect to the server using a putty.
> > > It is VM running in Amazon cloud.
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, 19 February, 2014 12:32 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Any issue with the network?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I just came from vacation.
> > > > When I tested it, I get a Connect Exception. The complete stack
> > > > trace is given below.
> > > >
> > > > Exception Stack Trace
> > > >
> > > > Java Thin Client:
> > > >
> > > > ava.net.ConnectException: Connection timed out: connect
> > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at
> kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > at
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> )
> > > > at
> > > >
> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > nc
> > > Pr
> > > oducer.scala:68)
> > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > > at
> > > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > > at
> > > >
> > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.
> > > sc
> > > al
> > > a:82)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$m
> > > cV
> > > $s
> > > p(DefaultEventHandler.scala:67)
> > > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > sc
> > > ala:67)
> > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > at
> > > >
> > > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > java:48)
> > > > at
> > > >
> > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer
> > > (K
> > > af
> > > kaProducerTest.java:33)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > at
> > > >
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
> > > pl
> > > .j
> > > ava:57)
> > > > at
> > > >
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
> > > ce
> > > ss
> > > orImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > at
> junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.r
> > > un
> > > (J
> > > Unit3TestReference.java:131)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > java:38)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > mo
> > > te
> > > TestRunner.java:467)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > mo
> > > te
> > > TestRunner.java:683)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteT
> > > es
> > > tR
> > > unner.java:390)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remote
> > > Te
> > > st
> > > Runner.java:197)
> > > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata
> > > > with correlation id 0 for topics [Set(mytopic)] from broker
> > > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > > >
> > > > Kafka Server:
> > > > [2014-02-19 03:38:22,587] ERROR
> > > > [Controller-3-to-broker-3-send-thread],
> > > > Controller 3's connection to broker
> > > > id:3,host:54.xx.xx.xx,port:10094 was unsuccessful
> > > > (kafka.controller.RequestSendThread)
> > > > java.net.ConnectException: Connection timed out
> > > > at sun.nio.ch.Net.connect(Native Method)
> > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at
> > > >
> > > kafka.controller.RequestSendThread.connectToBroker(ControllerChann
> > > el
> > > Ma
> > > nager.scala:172)
> > > > at
> > > >
> > > kafka.controller.RequestSendThread.<init>(ControllerChannelManager
> > > .s
> > > ca
> > > la:115)
> > > > at
> > > >
> > > kafka.controller.ControllerChannelManager.kafka$controller$Control
> > > le
> > > rC
> > > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > > at
> > > >
> > > kafka.controller.ControllerChannelManager.addBroker(ControllerChan
> > > ne
> > > lM
> > > anager.scala:65)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > $h
> > > an
> > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine
> > > .s
> > > ca
> > > la:265)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > $h
> > > an
> > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine
> > > .s
> > > ca
> > > la:265)
> > > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > $h
> > > an
> > > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > $h
> > > an
> > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun
> > > $h
> > > an
> > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleCh
> > > il
> > > dC
> > > hange(ReplicaStateMachine.scala:255)
> > > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > > at
> > > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker
> > > > startup
> > > callback
> > > > for 3 (kafka.controller.KafkaController)
> > > > [2014-02-19 03:38:22,592] INFO [Replica state machine on
> > > > controller
> 3]:
> > > > Invoking state change to OnlineReplica for replicas
> > > > (kafka.controller.ReplicaStateMachine)
> > > > [2014-02-19 03:38:22,595] INFO
> > > > [Controller-3-to-broker-3-send-thread],
> > > > Starting (kafka.controller.RequestSendThread)
> > > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on
> > > > Controller
> 3]:
> > > > Broker change listener fired for path /brokers/ids with children
> > > > 3,2,1
> > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on
> > > > Controller
> 3]:
> > > > Newly added brokers: 2,1, deleted brokers: , all live brokers:
> > > > 3,2,1
> > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > > balasubramanian.jayaraman@autodesk.com]
> > > > Sent: Thursday, February 06, 2014 5:40 PM
> > > > To: users@kafka.apache.org
> > > > Subject: RE: Reg Exception in Kafka
> > > >
> > > > After correcting the spelling, it is registered to correct public IP.
> > > >
> > > > Thanks
> > > > Bala
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > I think you mis-spelled the property name. It should be "
> > > > advertised.host.name", instead of advertise.host.name
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > Thanks Joel.
> > > > >
> > > > > It seems the Broker is registered to the zookeeper with the IP
> > > > > 10.199.31.87. The output pf the command is given below.
> > > > >
> > > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181
> > > > > get
> > > > > /brokers/ids/1
> > > > > Connecting to 0.0.0.0:2181
> > > > >
> > > > > WATCHER::
> > > > >
> > > > > WatchedEvent state:SyncConnected type:None path:null
> > > > >
> > > > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > > vers
> > > > > ion":1,"port":9092}
> > > > > cZxid = 0x2d
> > > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu
> > > > > Feb
> > > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength =
> > > > > 89
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Use zookeeper-shell script:
> > > > >
> > > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > > /brokers/ids/<your broker id>
> > > > >
> > > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > > Jayaraman
> > > > > (Contingent) wrote:
> > > > > > Where should I look for these information. From the logs, I
> > > > > > could see
> > > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka
> > > > > Server is started in port 9082 and bind to IP 10.x.x.x.
> > > > > > If I don't give the host.name in server.properties, I get "
> > > > > java.nio.channels.UnresolvedAddressException" and If I give
> > > > > the
> > > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > > It is the same behavior as in 0.8.0.
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > > http://10.199.31.87:9094/>)
> > > > > > is still the local ip, not the public one. Could you check
> > > > > > the broker registration in zk (
> > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+s
> > > > > tr
> > > > > uc
> > > > > ture
> > > > > s+in+Zookeeper
> > > > > )
> > > > > > and see what's the host/port of the broker?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I downloaded from the trunk and set up the properties
> > > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > > >
> > > > > > > Even after this change, I get the ConnectException. The
> > > > > > > details logs
> > > > > are
> > > > > > > given below. Is there any workaround for this?
> > > > > > >
> > > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > > 54.241.44.129:9094
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully
> > > > > > > fetched metadata
> > > > > for 1
> > > > > > > topic(s) Set(mytopic)
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > > partition info
> > > > > for
> > > > > > > topic mytopic
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition
> > > > > > > [mytopic,0] has
> > > > > leader 3
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > > registered for
> > > > > > > topic: mytopic are 0
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages
> > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > 2014-02-04 17:46:01,833] Producer sending messages with
> > > > > > > correlation id
> > > > > > > 2 for topics [mytopic,0] to broker 3 on
> > > > > 10.199.31.87:9094
> > > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection
> > > > > > > to
> > > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > > at
> > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:66
> > > > > 6)
> > > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > )
> > > > > > > at
> > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > at
> > > > > > >
> > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.sca
> > > > la
> > > > :1
> > > > 61)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSen
> > > > > d(
> > > > > Sy
> > > > > ncPr
> > > > > oducer.scala:68)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > $s
> > > > > p$
> > > > > 1.ap
> > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > $s
> > > > > p$
> > > > > 1.ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV
> > > > > $s
> > > > > p$
> > > > > 1.ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> > > > > > >
> > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > scala:101)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > .s
> > > > > ca
> > > > > la:1
> > > > > 01)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > .s
> > > > > ca
> > > > > la:1
> > > > > 01)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$
> > > > > De
> > > > > fa
> > > > > ultE
> > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > al
> > > > > iz
> > > > > edDa
> > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > al
> > > > > iz
> > > > > edDa
> > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > scal
> > > > > a:80)
> > > > > > > at
> > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > scal
> > > > > a:80)
> > > > > > > at
> > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > at
> > > > > > >
> > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.sca
> > > > la
> > > > :1
> > > > 61)
> > > > > > > at
> > > > > > >
> > > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > 194)
> > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > )
> > > > > > > at
> > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedDat
> > > > > a(
> > > > > De
> > > > > faul
> > > > > tEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHa
> > > > > nd
> > > > > le
> > > > > r.sc
> > > > > ala:72)
> > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > at
> > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > at
> > > > > > >
> > > > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > > java:48)
> > > > > > > at
> > > > > > >
> > > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProd
> > > > > uc
> > > > > er
> > > > > (Kaf
> > > > > kaProducerTest.java:33)
> > > > > > > at
> > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > Method)
> > > > > > > at
> > > > > > >
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccess
> > > > > or
> > > > > Im
> > > > > pl.j
> > > > > ava:57)
> > > > > > > at
> > > > > > >
> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMeth
> > > > > od
> > > > > Ac
> > > > > cess
> > > > > orImpl.java:43)
> > > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > > at
> > > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > > at
> > > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > > at
> junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReferen
> > > > > ce
> > > > > .r
> > > > > un(J
> > > > > Unit3TestReference.java:131)
> > > > > > > at
> > > > > > >
> > > > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > > java:38)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTest
> > > > > s(
> > > > > Re
> > > > > mote
> > > > > TestRunner.java:467)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTest
> > > > > s(
> > > > > Re
> > > > > mote
> > > > > TestRunner.java:683)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(Rem
> > > > > ot
> > > > > eT
> > > > > estR
> > > > > unner.java:390)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Re
> > > > > mo
> > > > > te
> > > > > Test
> > > > > Runner.java:197)
> > > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send
> > > > > > > producer request with correlation id 2 to broker 3 with
> > > > > > > data for partitions [mytopic,0]
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > > > Could
> > > > > some
> > > > > > > EC2 users here help out?
> > > > > > >
> > > > > > > Also, we recently added
> > > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > > which allows one to use a different ip for binding and
> > connecting.
> > > > > > > You
> > > > > can
> > > > > > > see if this works for you. The patch is only in trunk though.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian
> > > > > > > Jayaraman
> > > > > (Contingent) <
> > > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > > > Actually the public IP is not one of the IP configured
> > > > > > > > in the Ethernet
> > > > > interfaces.
> > > > > > > > Only the Local IP is configured in eth0.
> > > > > > > > Is there any solution to this?
> > > > > > > >
> > > > > > > > Ifconfig O/P:
> > > > > > > >
> > > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > > Mask:255.255.255.192
> > > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> > frame:0
> > > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > > carrier:0
> > > > > > > > collisions:0 txqueuelen:1000
> > > > > > > > RX bytes:40233350 (38.3 MiB) TX
> > > > > > > > bytes:15089154
> > > > > > > > (14.3
> > > > MiB)
> > > > > > > > Interrupt:25
> > > > > > > >
> > > > > > > > lo Link encap:Local Loopback
> > > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > > RX packets:1379711 errors:0 dropped:0
> > > > > > > > overruns:0
> > > frame:0
> > > > > > > > TX packets:1379711 errors:0 dropped:0
> > > > > > > > overruns:0
> > > > carrier:0
> > > > > > > > collisions:0 txqueuelen:0
> > > > > > > > RX bytes:109133672 (104.0 MiB) TX
> > > > > > > > bytes:109133672
> > > > > > > > (104.0
> > > > > > > > MiB)
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > Could it be a port conflict?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian
> > > > > > > > Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > Jun,
> > > > > > > > >
> > > > > > > > > Thanks for your help.
> > > > > > > > > I get the following exception :
> > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > bind to
> > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > la
> > > > > > > > :1
> > > > > > > > 88)
> > > > > > > > > at
> > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > at
> > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > at
> > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable
> > > > > .s
> > > > > ca
> > > > > la:3
> > > > > 4)
> > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > ja
> > > > > va:1
> > > > > 39)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > ja
> > > > > > > > va
> > > > > > > > :77)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > ja
> > > > > > > > va
> > > > > > > > :70)
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > la
> > > > > > > > :1
> > > > > > > > 84)
> > > > > > > > > ... 6 more
> > > > > > > > >
> > > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > > >
> > > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > > overridden to
> > > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is
> > > > > > > > > overridden to
> > > > > > > > > 9092
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads
> > > > > > > > > is
> > > > > overridden
> > > > > > > > > to 2
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > > overridden to
> > > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > log.cleanup.interval.mins
> > > > > is
> > > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > zookeeper.connect is overridden to
> > > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > > overridden
> > > > > to 1
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > num.network.threads is overridden to 2
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > zookeeper.connection.timeout.msis overridden to
> > > > > > > > > 1000000
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions
> > > > > > > > > is
> > > > > overridden
> > > > > > > > > to 2
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,182] INFO Property
> > > > > > > > > log.segment.bytes is overridden to
> > > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1],
> > > > > > > > > Starting
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker
> > > > > > > > > 1] Starting log cleaner every 60000 ms
> > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker
> > > > > > > > > 1] Starting log flusher every 3000 ms with the
> > > > > > > > > following overrides Map()
> > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > > KafkaServerStable
> > > > > > > > > startup. Prepare to shutdown
> > > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > bind to
> > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > la
> > > > > > > > :1
> > > > > > > > 88)
> > > > > > > > > at
> > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > at
> > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > at
> > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable
> > > > > .s
> > > > > ca
> > > > > la:3
> > > > > 4)
> > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > ja
> > > > > va:1
> > > > > 39)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > ja
> > > > > > > > va
> > > > > > > > :77)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.
> > > > > > > > ja
> > > > > > > > va
> > > > > > > > :70)
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.sca
> > > > > > > > la
> > > > > > > > :1
> > > > > > > > 84)
> > > > > > > > > ... 6 more
> > > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1],
> > > > > > > > > Shutting down
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on
> > > > > > > > > Broker 1],
> > > > > Shutting
> > > > > > > > > down
> > > > > > > > > (kafka.network.SocketServer)
> > > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on
> > > > > > > > > Broker 1],
> > > > > Shutdown
> > > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka
> > > > > > > > > scheduler
> > > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut
> > > > > > > > > down
> > > > > completed
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1],
> > > > > > > > > Shutting down
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > You should should use the public IP for host.name.
> > > > > > > > > What's the
> > > > > error
> > > > > > > > > you see during broker startup?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > I checked the faq. I did change the host.name in
> > > > > > > > > > server
> > > > > properties.
> > > > > > > > > > After changing it I get ConnectException.
> > > > > > > > > >
> > > > > > > > > > The problem here is in EC2 have a different public
> > > > > > > > > > IP address
> > > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > > which is 10.x.x.x. I think because of this there is
> > > > > > > > > > a
> > > > ConnectException.
> > > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > > >
> > > > > > > > > > What should be the IP address that is to be given in
> > > > > > > host.nameproperty.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > > >
> > > > > > > > > > Have you looked at
> > > > > > > > > >
> > > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC
> > > > > 2,
> > > > > wh
> > > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > > ?
> > > > > > > > > >
> > > > > > > > > > thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > > Jayaraman
> > > > > > > > > > (Contingent) <
> > > > > > > > > > balasubramanian.jayaraman@autodesk.com>
> > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I have a remote server (EC2) setup with Kafka
> > > > > > > > > > > cluster
> > > setup.
> > > > > > > > > > > There are
> > > > > > > > > > > 3 brokers each running in the port 9092,9093,9094.
> > > > > > > > > > > The
> > > > > zookeeper
> > > > > > > > > > > is running in the port 2181.
> > > > > > > > > > > When I send message to the brokers from my PC, I
> > > > > > > > > > > get an exception which is given below. I did a
> > > > > > > > > > > dump in the remote server, the request is received
> > > > > > > > > > > in the remote
> > server.
> > > > > > > > > > > I am able to locally test the consumer/producer
> > > > > > > > > > > script present in the bin folder. What am I missing?
> > > > > > > > > > > Can you kindly help me in this
> > > > > > > > > error?
> > > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > > >
> > > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:
> > > > > > > > > > > 90
> > > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > request.required.acks is overridden to 1 [ INFO]
> > > > > > > > > > > [main
> > > > > > > > > > > 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > key.serializer.class is overridden to
> > > > > > > > > > > kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class
> > > > > > > > > > > is
> > > > > overridden
> > > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main
> > > > > > > > > > > 2014-01-27 16:06:50,154]
> > > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main
> > > > > > > > > > > 2014-01-27 16:06:50,298] Handling 1 events [ INFO]
> > > > > > > > > > > [main
> > > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from
> > > > > > > > > > > broker
> > > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation
> > > > > > > > > > > id
> > > > > > > > > > > 0 for 1
> > > > > > > > > > > topic(s)
> > > > > > > > > > > Set(mytopic)
> > > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created
> > > > > > > > > > > socket with SO_TIMEOUT =
> > > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192
> > > > > > > > > > > (requested -1), SO_SNDBUF =
> > > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected
> > > > > > > > > > > to 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > > 54.241.44.129:9093 [DEBUG] [main 2014-01-27
> > > > > > > > > > > 15:59:44,025] Successfully fetched metadata for 1
> > > > > > > > > > > topic(s) Set(mytopic) [DEBUG] [main 2014-01-27
> > > > > > > > > > > 15:59:44,058] Getting broker partition info for
> > > > > > > > > > > topic mytopic [DEBUG] [main
> > > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072]
> > > > > > > > > > > Broker partitions registered for
> > > > > > > > > > > topic: mytopic are 0 [DEBUG] [main 2014-01-27
> > > > > > > > > > > 15:59:44,091] Sending 1 messages with no
> > > > > > > > > > > compression to [mytopic,0] [DEBUG] [main
> > > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > > with correlation id 2 for topics [mytopic,0] to
> > > > > > > > > > > broker
> > > > > > > > > > > 2 on
> > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > > connection to
> > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > unsuccessful
> java.nio.channels.UnresolvedAddressException
> > > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > > at
> > > > > > > > > >
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > > 640)
> > > > > > > > > > > at
> > > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:57
> > > > > > > > > )
> > > > > > > > > > > at
> > > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146
> > > > > > > )
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > >
> > > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.s
> > > > > ca
> > > > > la
> > > > > :161
> > > > > > > > )
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSen
> > > > > d(
> > > > > Sy
> > > > > > > > > > nc
> > > > > > > > > > Pr
> > > > > > > > > > oducer.scala:68)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > ap
> > > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > ap
> > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > ap
> > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > at
> > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > > scala:101)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > .s
> > > > > ca
> > > > > la
> > > > > > > > > :1
> > > > > > > > > 01)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer
> > > > > .s
> > > > > ca
> > > > > la
> > > > > > > > > :1
> > > > > > > > > 01)
> > > > > > > > > > > at
> > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > at
> > > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$
> > > > > De
> > > > > fa
> > > > > > > > > > ul
> > > > > > > > > > tE
> > > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > al
> > > > > iz
> > > > > > > > > > ed
> > > > > > > > > > Da
> > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSeri
> > > > > al
> > > > > iz
> > > > > > > > > > ed
> > > > > > > > > > Da
> > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > sc
> > > > > > > > > > al
> > > > > > > > > > a:80)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > sc
> > > > > > > > > > al
> > > > > > > > > > a:80)
> > > > > > > > > > > at
> > > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631
> > > > > > > )
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > >
> > > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.s
> > > > > ca
> > > > > la
> > > > > :161
> > > > > > > > )
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > > 194)
> > > > > > > > > > > at
> > > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:39
> > > > > > > > > )
> > > > > > > > > > > at
> > > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedDat
> > > > > a(
> > > > > De
> > > > > > > > > > fa
> > > > > > > > > > ul
> > > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > > sc
> > > > > > > > > > ala:72)
> > > > > > > > > > > at
> kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > > at
> > > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java
> > > > > :3
> > > > > 9)
> > > > > > > > > > > at
> > > > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.
> > > > > > > > > > > ja
> > > > > > > > > > > va:2
> > > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed
> > > > > > > > > > > to send producer request with correlation id 2 to
> > > > > > > > > > > broker
> > > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Bala
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: Reg Exception in Kafka
Posted by Jun Rao <ju...@gmail.com>.
Maybe you need to add the port to the right security group?
Thanks,
Jun
On Thu, Feb 20, 2014 at 9:58 PM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> One point to note is that, I am trying to access the Kafka broker (located
> in Amazon Cloud, EC2 instance) from the Eclipse (located in my office). I
> am using Kafka from the trunk .
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, 21 February, 2014 1:51 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> That explains why your producer hits connection timeout. Not sure why the
> controller to broker connection also times out though, if you can manually
> establish the connection.
>
> Thanks,
>
> Jun
>
>
> On Thu, Feb 20, 2014 at 7:37 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > All the brokers reside in the same server and are listening on the
> > port 10092,10093,10094. From the same machine I can connect to the
> > zookeeper and the brokers. But When I tried to connect from an
> > external machine (from Eclipse), I get an exception as communicated
> > earlier. I was not able to connect to any of the brokers. I get the
> > same exception while connecting to all the brokers.
> >
> > Regards
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, 20 February, 2014 12:05 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Can you make the connection from the controller host to the other broker?
> > Also, what's the # open file handlers on each broker? Do you see any
> > "too many open file handler" error in the broker log?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I don't think so. I am able to connect to the server using a putty.
> > > It is VM running in Amazon cloud.
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, 19 February, 2014 12:32 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Any issue with the network?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I just came from vacation.
> > > > When I tested it, I get a Connect Exception. The complete stack
> > > > trace is given below.
> > > >
> > > > Exception Stack Trace
> > > >
> > > > Java Thin Client:
> > > >
> > > > ava.net.ConnectException: Connection timed out: connect
> > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at
> kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > at
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > > at
> > > >
> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sync
> > > Pr
> > > oducer.scala:68)
> > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > > at
> > > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > > at
> > > >
> > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.sc
> > > al
> > > a:82)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV
> > > $s
> > > p(DefaultEventHandler.scala:67)
> > > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > > at
> > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > sc
> > > ala:67)
> > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > at
> > > >
> > > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > java:48)
> > > > at
> > > >
> > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(K
> > > af
> > > kaProducerTest.java:33)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > at
> > > >
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
> > > .j
> > > ava:57)
> > > > at
> > > >
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> > > ss
> > > orImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > at
> junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run
> > > (J
> > > Unit3TestReference.java:131)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > java:38)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remo
> > > te
> > > TestRunner.java:467)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remo
> > > te
> > > TestRunner.java:683)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTes
> > > tR
> > > unner.java:390)
> > > > at
> > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTe
> > > st
> > > Runner.java:197)
> > > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata
> > > > with correlation id 0 for topics [Set(mytopic)] from broker
> > > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > > >
> > > > Kafka Server:
> > > > [2014-02-19 03:38:22,587] ERROR
> > > > [Controller-3-to-broker-3-send-thread],
> > > > Controller 3's connection to broker
> > > > id:3,host:54.xx.xx.xx,port:10094 was unsuccessful
> > > > (kafka.controller.RequestSendThread)
> > > > java.net.ConnectException: Connection timed out
> > > > at sun.nio.ch.Net.connect(Native Method)
> > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at
> > > >
> > > kafka.controller.RequestSendThread.connectToBroker(ControllerChannel
> > > Ma
> > > nager.scala:172)
> > > > at
> > > >
> > > kafka.controller.RequestSendThread.<init>(ControllerChannelManager.s
> > > ca
> > > la:115)
> > > > at
> > > >
> > > kafka.controller.ControllerChannelManager.kafka$controller$Controlle
> > > rC
> > > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > > at
> > > >
> > > kafka.controller.ControllerChannelManager.addBroker(ControllerChanne
> > > lM
> > > anager.scala:65)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > > an
> > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.s
> > > ca
> > > la:265)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > > an
> > > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.s
> > > ca
> > > la:265)
> > > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > > an
> > > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > > an
> > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > > an
> > > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChil
> > > dC
> > > hange(ReplicaStateMachine.scala:255)
> > > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > > at
> > > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup
> > > callback
> > > > for 3 (kafka.controller.KafkaController)
> > > > [2014-02-19 03:38:22,592] INFO [Replica state machine on controller
> 3]:
> > > > Invoking state change to OnlineReplica for replicas
> > > > (kafka.controller.ReplicaStateMachine)
> > > > [2014-02-19 03:38:22,595] INFO
> > > > [Controller-3-to-broker-3-send-thread],
> > > > Starting (kafka.controller.RequestSendThread)
> > > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller
> 3]:
> > > > Broker change listener fired for path /brokers/ids with children
> > > > 3,2,1
> > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller
> 3]:
> > > > Newly added brokers: 2,1, deleted brokers: , all live brokers:
> > > > 3,2,1
> > > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > > balasubramanian.jayaraman@autodesk.com]
> > > > Sent: Thursday, February 06, 2014 5:40 PM
> > > > To: users@kafka.apache.org
> > > > Subject: RE: Reg Exception in Kafka
> > > >
> > > > After correcting the spelling, it is registered to correct public IP.
> > > >
> > > > Thanks
> > > > Bala
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > I think you mis-spelled the property name. It should be "
> > > > advertised.host.name", instead of advertise.host.name
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > Thanks Joel.
> > > > >
> > > > > It seems the Broker is registered to the zookeeper with the IP
> > > > > 10.199.31.87. The output pf the command is given below.
> > > > >
> > > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181
> > > > > get
> > > > > /brokers/ids/1
> > > > > Connecting to 0.0.0.0:2181
> > > > >
> > > > > WATCHER::
> > > > >
> > > > > WatchedEvent state:SyncConnected type:None path:null
> > > > >
> > > > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > > vers
> > > > > ion":1,"port":9092}
> > > > > cZxid = 0x2d
> > > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu
> > > > > Feb
> > > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength =
> > > > > 89
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Use zookeeper-shell script:
> > > > >
> > > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > > /brokers/ids/<your broker id>
> > > > >
> > > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > > Jayaraman
> > > > > (Contingent) wrote:
> > > > > > Where should I look for these information. From the logs, I
> > > > > > could see
> > > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server
> > > > > is started in port 9082 and bind to IP 10.x.x.x.
> > > > > > If I don't give the host.name in server.properties, I get "
> > > > > java.nio.channels.UnresolvedAddressException" and If I give the
> > > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > > It is the same behavior as in 0.8.0.
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > > http://10.199.31.87:9094/>)
> > > > > > is still the local ip, not the public one. Could you check the
> > > > > > broker registration in zk (
> > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+str
> > > > > uc
> > > > > ture
> > > > > s+in+Zookeeper
> > > > > )
> > > > > > and see what's the host/port of the broker?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I downloaded from the trunk and set up the properties
> > > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > > >
> > > > > > > Even after this change, I get the ConnectException. The
> > > > > > > details logs
> > > > > are
> > > > > > > given below. Is there any workaround for this?
> > > > > > >
> > > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > > 54.241.44.129:9094
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > > > > metadata
> > > > > for 1
> > > > > > > topic(s) Set(mytopic)
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > > partition info
> > > > > for
> > > > > > > topic mytopic
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0]
> > > > > > > has
> > > > > leader 3
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > > registered for
> > > > > > > topic: mytopic are 0
> > > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages
> > > > > > > with no compression to [mytopic,0] [DEBUG] [main 2014-02-04
> > > > > > > 17:46:01,833] Producer sending messages with correlation id
> > > > > > > 2 for topics [mytopic,0] to broker 3 on
> > > > > 10.199.31.87:9094
> > > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection
> > > > > > > to
> > > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > > at
> > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > > at
> > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > at
> > > > > > >
> > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala
> > > > :1
> > > > 61)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(
> > > > > Sy
> > > > > ncPr
> > > > > oducer.scala:68)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > > p$
> > > > > 1.ap
> > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > > p$
> > > > > 1.ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > > p$
> > > > > 1.ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> > > > > > >
> > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > scala:101)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > > ca
> > > > > la:1
> > > > > 01)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > > ca
> > > > > la:1
> > > > > 01)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$De
> > > > > fa
> > > > > ultE
> > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > > iz
> > > > > edDa
> > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > > iz
> > > > > edDa
> > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > scal
> > > > > a:80)
> > > > > > > at
> > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > scal
> > > > > a:80)
> > > > > > > at
> > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > at
> > > > > > >
> > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala
> > > > :1
> > > > 61)
> > > > > > > at
> > > > > > >
> > > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > 194)
> > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > > at
> > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(
> > > > > De
> > > > > faul
> > > > > tEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHand
> > > > > le
> > > > > r.sc
> > > > > ala:72)
> > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > at
> > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > at
> > > > > > >
> > > > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > > java:48)
> > > > > > > at
> > > > > > >
> > > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProduc
> > > > > er
> > > > > (Kaf
> > > > > kaProducerTest.java:33)
> > > > > > > at
> > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > > Method)
> > > > > > > at
> > > > > > >
> > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor
> > > > > Im
> > > > > pl.j
> > > > > ava:57)
> > > > > > > at
> > > > > > >
> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
> > > > > Ac
> > > > > cess
> > > > > orImpl.java:43)
> > > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > > at
> > > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > > at
> > > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > > at
> junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference
> > > > > .r
> > > > > un(J
> > > > > Unit3TestReference.java:131)
> > > > > > > at
> > > > > > >
> > > > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > > java:38)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(
> > > > > Re
> > > > > mote
> > > > > TestRunner.java:467)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(
> > > > > Re
> > > > > mote
> > > > > TestRunner.java:683)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(Remot
> > > > > eT
> > > > > estR
> > > > > unner.java:390)
> > > > > > > at
> > > > > > >
> > > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remo
> > > > > te
> > > > > Test
> > > > > Runner.java:197)
> > > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send
> > > > > > > producer request with correlation id 2 to broker 3 with data
> > > > > > > for partitions [mytopic,0]
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > > > Could
> > > > > some
> > > > > > > EC2 users here help out?
> > > > > > >
> > > > > > > Also, we recently added
> > > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > > which allows one to use a different ip for binding and
> > connecting.
> > > > > > > You
> > > > > can
> > > > > > > see if this works for you. The patch is only in trunk though.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > > > > (Contingent) <
> > > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > > > Actually the public IP is not one of the IP configured in
> > > > > > > > the Ethernet
> > > > > interfaces.
> > > > > > > > Only the Local IP is configured in eth0.
> > > > > > > > Is there any solution to this?
> > > > > > > >
> > > > > > > > Ifconfig O/P:
> > > > > > > >
> > > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > > Mask:255.255.255.192
> > > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> > frame:0
> > > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > > carrier:0
> > > > > > > > collisions:0 txqueuelen:1000
> > > > > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154
> > > > > > > > (14.3
> > > > MiB)
> > > > > > > > Interrupt:25
> > > > > > > >
> > > > > > > > lo Link encap:Local Loopback
> > > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > > RX packets:1379711 errors:0 dropped:0 overruns:0
> > > frame:0
> > > > > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> > > > carrier:0
> > > > > > > > collisions:0 txqueuelen:0
> > > > > > > > RX bytes:109133672 (104.0 MiB) TX
> > > > > > > > bytes:109133672
> > > > > > > > (104.0
> > > > > > > > MiB)
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > Could it be a port conflict?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > Jun,
> > > > > > > > >
> > > > > > > > > Thanks for your help.
> > > > > > > > > I get the following exception :
> > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > bind to
> > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > > :1
> > > > > > > > 88)
> > > > > > > > > at
> > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > at
> > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > at
> > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.s
> > > > > ca
> > > > > la:3
> > > > > 4)
> > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > ja
> > > > > va:1
> > > > > 39)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > > va
> > > > > > > > :77)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > > va
> > > > > > > > :70)
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > > :1
> > > > > > > > 84)
> > > > > > > > > ... 6 more
> > > > > > > > >
> > > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > > >
> > > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > > overridden to
> > > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is
> > > > > > > > > overridden to
> > > > > > > > > 9092
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads
> > > > > > > > > is
> > > > > overridden
> > > > > > > > > to 2
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > > overridden to
> > > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > log.cleanup.interval.mins
> > > > > is
> > > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > > zookeeper.connect is overridden to
> > > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > > overridden
> > > > > to 1
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > num.network.threads is overridden to 2
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions
> > > > > > > > > is
> > > > > overridden
> > > > > > > > > to 2
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,182] INFO Property
> > > > > > > > > log.segment.bytes is overridden to
> > > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1],
> > > > > > > > > Starting
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > > > > Starting log cleaner every 60000 ms
> > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > > > > Starting log flusher every 3000 ms with the following
> > > > > > > > > overrides Map()
> > > > > > > > > (kafka.log.LogManager)
> > > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > > KafkaServerStable
> > > > > > > > > startup. Prepare to shutdown
> > > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > > bind to
> > > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > > :1
> > > > > > > > 88)
> > > > > > > > > at
> > > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > > at
> > > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > > at
> > > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.s
> > > > > ca
> > > > > la:3
> > > > > 4)
> > > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > > ja
> > > > > va:1
> > > > > 39)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > > va
> > > > > > > > :77)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > > va
> > > > > > > > :70)
> > > > > > > > > at
> > > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > > :1
> > > > > > > > 84)
> > > > > > > > > ... 6 more
> > > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1],
> > > > > > > > > Shutting down
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker
> > > > > > > > > 1],
> > > > > Shutting
> > > > > > > > > down
> > > > > > > > > (kafka.network.SocketServer)
> > > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker
> > > > > > > > > 1],
> > > > > Shutdown
> > > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut
> > > > > > > > > down
> > > > > completed
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1],
> > > > > > > > > Shutting down
> > > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > >
> > > > > > > > > Regards
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > You should should use the public IP for host.name.
> > > > > > > > > What's the
> > > > > error
> > > > > > > > > you see during broker startup?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > I checked the faq. I did change the host.name in
> > > > > > > > > > server
> > > > > properties.
> > > > > > > > > > After changing it I get ConnectException.
> > > > > > > > > >
> > > > > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > > > > address
> > > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > > which is 10.x.x.x. I think because of this there is a
> > > > ConnectException.
> > > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > > >
> > > > > > > > > > What should be the IP address that is to be given in
> > > > > > > host.nameproperty.
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > > > -----Original Message-----
> > > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > > >
> > > > > > > > > > Have you looked at
> > > > > > > > > >
> > > > > > > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,
> > > > > wh
> > > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > > ?
> > > > > > > > > >
> > > > > > > > > > thanks,
> > > > > > > > > >
> > > > > > > > > > Jun
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > > Jayaraman
> > > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > I have a remote server (EC2) setup with Kafka
> > > > > > > > > > > cluster
> > > setup.
> > > > > > > > > > > There are
> > > > > > > > > > > 3 brokers each running in the port 9092,9093,9094.
> > > > > > > > > > > The
> > > > > zookeeper
> > > > > > > > > > > is running in the port 2181.
> > > > > > > > > > > When I send message to the brokers from my PC, I get
> > > > > > > > > > > an exception which is given below. I did a dump in
> > > > > > > > > > > the remote server, the request is received in the
> > > > > > > > > > > remote
> > server.
> > > > > > > > > > > I am able to locally test the consumer/producer
> > > > > > > > > > > script present in the bin folder. What am I missing?
> > > > > > > > > > > Can you kindly help me in this
> > > > > > > > > error?
> > > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > > >
> > > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:
> > > > > > > > > > > 90
> > > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > request.required.acks is overridden to 1 [ INFO]
> > > > > > > > > > > [main
> > > > > > > > > > > 2014-01-27 16:06:50,108] Property
> > > > > > > > > > > key.serializer.class is overridden to
> > > > > > > > > > > kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class
> > > > > > > > > > > is
> > > > > overridden
> > > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main
> > > > > > > > > > > 2014-01-27 16:06:50,154]
> > > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from
> > > > > > > > > > > broker
> > > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation
> > > > > > > > > > > id
> > > > > > > > > > > 0 for 1
> > > > > > > > > > > topic(s)
> > > > > > > > > > > Set(mytopic)
> > > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created
> > > > > > > > > > > socket with SO_TIMEOUT =
> > > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested
> > > > > > > > > > > -1), SO_SNDBUF =
> > > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > > > > 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > > 54.241.44.129:9093
> > > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > > > > fetched metadata for 1
> > > > > > > > > > > topic(s) Set(mytopic) [DEBUG] [main 2014-01-27
> > > > > > > > > > > 15:59:44,058] Getting broker partition info for
> > > > > > > > > > > topic mytopic [DEBUG] [main
> > > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072]
> > > > > > > > > > > Broker partitions registered for
> > > > > > > > > > > topic: mytopic are 0 [DEBUG] [main 2014-01-27
> > > > > > > > > > > 15:59:44,091] Sending 1 messages with no compression
> > > > > > > > > > > to [mytopic,0] [DEBUG] [main
> > > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > > with correlation id 2 for topics [mytopic,0] to
> > > > > > > > > > > broker
> > > > > > > > > > > 2 on
> > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > > connection to
> > > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > > unsuccessful
> java.nio.channels.UnresolvedAddressException
> > > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > > at
> > > > > > > > > >
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > > 640)
> > > > > > > > > > > at
> > > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.sc
> > > > > > > > > al
> > > > > > > > > a:57
> > > > > > > > > )
> > > > > > > > > > > at
> > > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > >
> > > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.sca
> > > > > la
> > > > > :161
> > > > > > > > )
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(
> > > > > Sy
> > > > > > > > > > nc
> > > > > > > > > > Pr
> > > > > > > > > > oducer.scala:68)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > ap
> > > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > ap
> > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > > ap
> > > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > > at
> > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > > scala:101)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > > ca
> > > > > la
> > > > > > > > > :1
> > > > > > > > > 01)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > > ca
> > > > > la
> > > > > > > > > :1
> > > > > > > > > 01)
> > > > > > > > > > > at
> > > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > > at
> > > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$De
> > > > > fa
> > > > > > > > > > ul
> > > > > > > > > > tE
> > > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > > iz
> > > > > > > > > > ed
> > > > > > > > > > Da
> > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > > iz
> > > > > > > > > > ed
> > > > > > > > > > Da
> > > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > sc
> > > > > > > > > > al
> > > > > > > > > > a:80)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > > sc
> > > > > > > > > > al
> > > > > > > > > > a:80)
> > > > > > > > > > > at
> > > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > >
> > > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.sca
> > > > > la
> > > > > :161
> > > > > > > > )
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > > 194)
> > > > > > > > > > > at
> > > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.sc
> > > > > > > > > al
> > > > > > > > > a:39
> > > > > > > > > )
> > > > > > > > > > > at
> > > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(
> > > > > De
> > > > > > > > > > fa
> > > > > > > > > > ul
> > > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > > > > > >
> > > > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > > sc
> > > > > > > > > > ala:72)
> > > > > > > > > > > at
> kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > > at
> > > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > > at
> > > > > > > > > > >
> > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:3
> > > > > 9)
> > > > > > > > > > > at
> > > > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.
> > > > > > > > > > > ja
> > > > > > > > > > > va:2
> > > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to
> > > > > > > > > > > send producer request with correlation id 2 to
> > > > > > > > > > > broker
> > > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > > >
> > > > > > > > > > > Thanks
> > > > > > > > > > > Bala
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
One point to note is that, I am trying to access the Kafka broker (located in Amazon Cloud, EC2 instance) from the Eclipse (located in my office). I am using Kafka from the trunk .
Thanks
Bala
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, 21 February, 2014 1:51 PM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
That explains why your producer hits connection timeout. Not sure why the controller to broker connection also times out though, if you can manually establish the connection.
Thanks,
Jun
On Thu, Feb 20, 2014 at 7:37 PM, Balasubramanian Jayaraman (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> All the brokers reside in the same server and are listening on the
> port 10092,10093,10094. From the same machine I can connect to the
> zookeeper and the brokers. But When I tried to connect from an
> external machine (from Eclipse), I get an exception as communicated
> earlier. I was not able to connect to any of the brokers. I get the
> same exception while connecting to all the brokers.
>
> Regards
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, 20 February, 2014 12:05 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Can you make the connection from the controller host to the other broker?
> Also, what's the # open file handlers on each broker? Do you see any
> "too many open file handler" error in the broker log?
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman
> (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I don't think so. I am able to connect to the server using a putty.
> > It is VM running in Amazon cloud.
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, 19 February, 2014 12:32 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Any issue with the network?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I just came from vacation.
> > > When I tested it, I get a Connect Exception. The complete stack
> > > trace is given below.
> > >
> > > Exception Stack Trace
> > >
> > > Java Thin Client:
> > >
> > > ava.net.ConnectException: Connection timed out: connect
> > > at sun.nio.ch.Net.connect0(Native Method)
> > > at sun.nio.ch.Net.connect(Net.java:465)
> > > at sun.nio.ch.Net.connect(Net.java:457)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > at
> > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sync
> > Pr
> > oducer.scala:68)
> > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > at
> > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > at
> > >
> > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.sc
> > al
> > a:82)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV
> > $s
> > p(DefaultEventHandler.scala:67)
> > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > sc
> > ala:67)
> > > at kafka.producer.Producer.send(Producer.scala:76)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at
> > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > java:48)
> > > at
> > >
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(K
> > af
> > kaProducerTest.java:33)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl
> > .j
> > ava:57)
> > > at
> > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce
> > ss
> > orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > > at junit.framework.TestResult.run(TestResult.java:109)
> > > at junit.framework.TestCase.run(TestCase.java:118)
> > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run
> > (J
> > Unit3TestReference.java:131)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > java:38)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remo
> > te
> > TestRunner.java:467)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remo
> > te
> > TestRunner.java:683)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTes
> > tR
> > unner.java:390)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTe
> > st
> > Runner.java:197)
> > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata
> > > with correlation id 0 for topics [Set(mytopic)] from broker
> > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > >
> > > Kafka Server:
> > > [2014-02-19 03:38:22,587] ERROR
> > > [Controller-3-to-broker-3-send-thread],
> > > Controller 3's connection to broker
> > > id:3,host:54.xx.xx.xx,port:10094 was unsuccessful
> > > (kafka.controller.RequestSendThread)
> > > java.net.ConnectException: Connection timed out
> > > at sun.nio.ch.Net.connect(Native Method)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at
> > >
> > kafka.controller.RequestSendThread.connectToBroker(ControllerChannel
> > Ma
> > nager.scala:172)
> > > at
> > >
> > kafka.controller.RequestSendThread.<init>(ControllerChannelManager.s
> > ca
> > la:115)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.kafka$controller$Controlle
> > rC
> > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.addBroker(ControllerChanne
> > lM
> > anager.scala:65)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.s
> > ca
> > la:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.s
> > ca
> > la:265)
> > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$h
> > an
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChil
> > dC
> > hange(ReplicaStateMachine.scala:255)
> > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > at
> > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup
> > callback
> > > for 3 (kafka.controller.KafkaController)
> > > [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> > > Invoking state change to OnlineReplica for replicas
> > > (kafka.controller.ReplicaStateMachine)
> > > [2014-02-19 03:38:22,595] INFO
> > > [Controller-3-to-broker-3-send-thread],
> > > Starting (kafka.controller.RequestSendThread)
> > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> > > Broker change listener fired for path /brokers/ids with children
> > > 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> > > Newly added brokers: 2,1, deleted brokers: , all live brokers:
> > > 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > >
> > >
> > > -----Original Message-----
> > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > balasubramanian.jayaraman@autodesk.com]
> > > Sent: Thursday, February 06, 2014 5:40 PM
> > > To: users@kafka.apache.org
> > > Subject: RE: Reg Exception in Kafka
> > >
> > > After correcting the spelling, it is registered to correct public IP.
> > >
> > > Thanks
> > > Bala
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > I think you mis-spelled the property name. It should be "
> > > advertised.host.name", instead of advertise.host.name
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > Thanks Joel.
> > > >
> > > > It seems the Broker is registered to the zookeeper with the IP
> > > > 10.199.31.87. The output pf the command is given below.
> > > >
> > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181
> > > > get
> > > > /brokers/ids/1
> > > > Connecting to 0.0.0.0:2181
> > > >
> > > > WATCHER::
> > > >
> > > > WatchedEvent state:SyncConnected type:None path:null
> > > >
> > > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > vers
> > > > ion":1,"port":9092}
> > > > cZxid = 0x2d
> > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu
> > > > Feb
> > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength =
> > > > 89
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Use zookeeper-shell script:
> > > >
> > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > /brokers/ids/<your broker id>
> > > >
> > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > Jayaraman
> > > > (Contingent) wrote:
> > > > > Where should I look for these information. From the logs, I
> > > > > could see
> > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server
> > > > is started in port 9082 and bind to IP 10.x.x.x.
> > > > > If I don't give the host.name in server.properties, I get "
> > > > java.nio.channels.UnresolvedAddressException" and If I give the
> > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > It is the same behavior as in 0.8.0.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > http://10.199.31.87:9094/>)
> > > > > is still the local ip, not the public one. Could you check the
> > > > > broker registration in zk (
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+str
> > > > uc
> > > > ture
> > > > s+in+Zookeeper
> > > > )
> > > > > and see what's the host/port of the broker?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I downloaded from the trunk and set up the properties
> > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > >
> > > > > > Even after this change, I get the ConnectException. The
> > > > > > details logs
> > > > are
> > > > > > given below. Is there any workaround for this?
> > > > > >
> > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > 54.241.44.129:9094
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > > > metadata
> > > > for 1
> > > > > > topic(s) Set(mytopic)
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > partition info
> > > > for
> > > > > > topic mytopic
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0]
> > > > > > has
> > > > leader 3
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > registered for
> > > > > > topic: mytopic are 0
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages
> > > > > > with no compression to [mytopic,0] [DEBUG] [main 2014-02-04
> > > > > > 17:46:01,833] Producer sending messages with correlation id
> > > > > > 2 for topics [mytopic,0] to broker 3 on
> > > > 10.199.31.87:9094
> > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection
> > > > > > to
> > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > at
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala
> > > :1
> > > 61)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(
> > > > Sy
> > > > ncPr
> > > > oducer.scala:68)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > p$
> > > > 1.ap
> > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > p$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$s
> > > > p$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > scala:101)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la:1
> > > > 01)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la:1
> > > > 01)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$De
> > > > fa
> > > > ultE
> > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > at
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala
> > > :1
> > > 61)
> > > > > > at
> > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > 194)
> > > > > > at
> > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > at
> > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(
> > > > De
> > > > faul
> > > > tEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHand
> > > > le
> > > > r.sc
> > > > ala:72)
> > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > at
> > > > > >
> > > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > java:48)
> > > > > > at
> > > > > >
> > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProduc
> > > > er
> > > > (Kaf
> > > > kaProducerTest.java:33)
> > > > > > at
> > > > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> > > > > > at
> > > > > >
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor
> > > > Im
> > > > pl.j
> > > > ava:57)
> > > > > > at
> > > > > >
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod
> > > > Ac
> > > > cess
> > > > orImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > at
> > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > at
> > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference
> > > > .r
> > > > un(J
> > > > Unit3TestReference.java:131)
> > > > > > at
> > > > > >
> > > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > java:38)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(
> > > > Re
> > > > mote
> > > > TestRunner.java:467)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(
> > > > Re
> > > > mote
> > > > TestRunner.java:683)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(Remot
> > > > eT
> > > > estR
> > > > unner.java:390)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remo
> > > > te
> > > > Test
> > > > Runner.java:197)
> > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send
> > > > > > producer request with correlation id 2 to broker 3 with data
> > > > > > for partitions [mytopic,0]
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > > Could
> > > > some
> > > > > > EC2 users here help out?
> > > > > >
> > > > > > Also, we recently added
> > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > which allows one to use a different ip for binding and
> connecting.
> > > > > > You
> > > > can
> > > > > > see if this works for you. The patch is only in trunk though.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > > > (Contingent) <
> > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > > Actually the public IP is not one of the IP configured in
> > > > > > > the Ethernet
> > > > interfaces.
> > > > > > > Only the Local IP is configured in eth0.
> > > > > > > Is there any solution to this?
> > > > > > >
> > > > > > > Ifconfig O/P:
> > > > > > >
> > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > Mask:255.255.255.192
> > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> frame:0
> > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > carrier:0
> > > > > > > collisions:0 txqueuelen:1000
> > > > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154
> > > > > > > (14.3
> > > MiB)
> > > > > > > Interrupt:25
> > > > > > >
> > > > > > > lo Link encap:Local Loopback
> > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > RX packets:1379711 errors:0 dropped:0 overruns:0
> > frame:0
> > > > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> > > carrier:0
> > > > > > > collisions:0 txqueuelen:0
> > > > > > > RX bytes:109133672 (104.0 MiB) TX
> > > > > > > bytes:109133672
> > > > > > > (104.0
> > > > > > > MiB)
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Could it be a port conflict?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > Thanks for your help.
> > > > > > > > I get the following exception :
> > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > bind to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.s
> > > > ca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > >
> > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > >
> > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > overridden to
> > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is
> > > > > > > > overridden to
> > > > > > > > 9092
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads
> > > > > > > > is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > overridden to
> > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > log.cleanup.interval.mins
> > > > is
> > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > zookeeper.connect is overridden to
> > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > overridden
> > > > to 1
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > num.network.threads is overridden to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions
> > > > > > > > is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,182] INFO Property
> > > > > > > > log.segment.bytes is overridden to
> > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1],
> > > > > > > > Starting
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log cleaner every 60000 ms
> > > > > > > > (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log flusher every 3000 ms with the following
> > > > > > > > overrides Map()
> > > > > > > > (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > KafkaServerStable
> > > > > > > > startup. Prepare to shutdown
> > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > kafka.common.KafkaException: Socket server failed to
> > > > > > > > bind to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.s
> > > > ca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.
> > > > ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.ja
> > > > > > > va
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala
> > > > > > > :1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1],
> > > > > > > > Shutting down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutting
> > > > > > > > down
> > > > > > > > (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutdown
> > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut
> > > > > > > > down
> > > > completed
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1],
> > > > > > > > Shutting down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > You should should use the public IP for host.name.
> > > > > > > > What's the
> > > > error
> > > > > > > > you see during broker startup?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian
> > > > > > > > Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > I checked the faq. I did change the host.name in
> > > > > > > > > server
> > > > properties.
> > > > > > > > > After changing it I get ConnectException.
> > > > > > > > >
> > > > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > > > address
> > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > which is 10.x.x.x. I think because of this there is a
> > > ConnectException.
> > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > >
> > > > > > > > > What should be the IP address that is to be given in
> > > > > > host.nameproperty.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > Have you looked at
> > > > > > > > >
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,
> > > > wh
> > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > ?
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have a remote server (EC2) setup with Kafka
> > > > > > > > > > cluster
> > setup.
> > > > > > > > > > There are
> > > > > > > > > > 3 brokers each running in the port 9092,9093,9094.
> > > > > > > > > > The
> > > > zookeeper
> > > > > > > > > > is running in the port 2181.
> > > > > > > > > > When I send message to the brokers from my PC, I get
> > > > > > > > > > an exception which is given below. I did a dump in
> > > > > > > > > > the remote server, the request is received in the
> > > > > > > > > > remote
> server.
> > > > > > > > > > I am able to locally test the consumer/producer
> > > > > > > > > > script present in the bin folder. What am I missing?
> > > > > > > > > > Can you kindly help me in this
> > > > > > > > error?
> > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > >
> > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:
> > > > > > > > > > 90
> > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > request.required.acks is overridden to 1 [ INFO]
> > > > > > > > > > [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property
> > > > > > > > > > key.serializer.class is overridden to
> > > > > > > > > > kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class
> > > > > > > > > > is
> > > > overridden
> > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,154]
> > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from
> > > > > > > > > > broker
> > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation
> > > > > > > > > > id
> > > > > > > > > > 0 for 1
> > > > > > > > > > topic(s)
> > > > > > > > > > Set(mytopic)
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created
> > > > > > > > > > socket with SO_TIMEOUT =
> > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested
> > > > > > > > > > -1), SO_SNDBUF =
> > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > > > 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > 54.241.44.129:9093
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > > > fetched metadata for 1
> > > > > > > > > > topic(s) Set(mytopic) [DEBUG] [main 2014-01-27
> > > > > > > > > > 15:59:44,058] Getting broker partition info for
> > > > > > > > > > topic mytopic [DEBUG] [main
> > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072]
> > > > > > > > > > Broker partitions registered for
> > > > > > > > > > topic: mytopic are 0 [DEBUG] [main 2014-01-27
> > > > > > > > > > 15:59:44,091] Sending 1 messages with no compression
> > > > > > > > > > to [mytopic,0] [DEBUG] [main
> > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > with correlation id 2 for topics [mytopic,0] to
> > > > > > > > > > broker
> > > > > > > > > > 2 on
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > connection to
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > at
> > > > > > > > >
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > 640)
> > > > > > > > > > at
> > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.sc
> > > > > > > > al
> > > > > > > > a:57
> > > > > > > > )
> > > > > > > > > > at
> > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.sca
> > > > la
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(
> > > > Sy
> > > > > > > > > nc
> > > > > > > > > Pr
> > > > > > > > > oducer.scala:68)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > scala:101)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.s
> > > > ca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$De
> > > > fa
> > > > > > > > > ul
> > > > > > > > > tE
> > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerial
> > > > iz
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.sca
> > > > la
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > 194)
> > > > > > > > > > at
> > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.sc
> > > > > > > > al
> > > > > > > > a:39
> > > > > > > > )
> > > > > > > > > > at
> > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(
> > > > De
> > > > > > > > > fa
> > > > > > > > > ul
> > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > sc
> > > > > > > > > ala:72)
> > > > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > at
> > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:3
> > > > 9)
> > > > > > > > > > at
> > > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.
> > > > > > > > > > ja
> > > > > > > > > > va:2
> > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to
> > > > > > > > > > send producer request with correlation id 2 to
> > > > > > > > > > broker
> > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>
Re: Reg Exception in Kafka
Posted by Jun Rao <ju...@gmail.com>.
That explains why your producer hits connection timeout. Not sure why the
controller to broker connection also times out though, if you can manually
establish the connection.
Thanks,
Jun
On Thu, Feb 20, 2014 at 7:37 PM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> All the brokers reside in the same server and are listening on the port
> 10092,10093,10094. From the same machine I can connect to the zookeeper and
> the brokers. But When I tried to connect from an external machine (from
> Eclipse), I get an exception as communicated earlier. I was not able to
> connect to any of the brokers. I get the same exception while connecting to
> all the brokers.
>
> Regards
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, 20 February, 2014 12:05 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Can you make the connection from the controller host to the other broker?
> Also, what's the # open file handlers on each broker? Do you see any "too
> many open file handler" error in the broker log?
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I don't think so. I am able to connect to the server using a putty. It
> > is VM running in Amazon cloud.
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, 19 February, 2014 12:32 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Any issue with the network?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I just came from vacation.
> > > When I tested it, I get a Connect Exception. The complete stack
> > > trace is given below.
> > >
> > > Exception Stack Trace
> > >
> > > Java Thin Client:
> > >
> > > ava.net.ConnectException: Connection timed out: connect
> > > at sun.nio.ch.Net.connect0(Native Method)
> > > at sun.nio.ch.Net.connect(Net.java:465)
> > > at sun.nio.ch.Net.connect(Net.java:457)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > at
> > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> > oducer.scala:68)
> > > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > > at
> > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > > at
> > >
> > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scal
> > a:82)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$s
> > p(DefaultEventHandler.scala:67)
> > > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > > at
> > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> > ala:67)
> > > at kafka.producer.Producer.send(Producer.scala:76)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at
> > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > java:48)
> > > at
> > >
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> > kaProducerTest.java:33)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:57)
> > > at
> > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > > at junit.framework.TestResult.run(TestResult.java:109)
> > > at junit.framework.TestCase.run(TestCase.java:118)
> > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> > Unit3TestReference.java:131)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > java:38)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:467)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:683)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> > unner.java:390)
> > > at
> > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> > Runner.java:197)
> > > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata with
> > > correlation id 0 for topics [Set(mytopic)] from broker
> > > [id:2,host:54.xx.xx.xx,port:10094] failed
> > >
> > > Kafka Server:
> > > [2014-02-19 03:38:22,587] ERROR
> > > [Controller-3-to-broker-3-send-thread],
> > > Controller 3's connection to broker id:3,host:54.xx.xx.xx,port:10094
> > > was unsuccessful (kafka.controller.RequestSendThread)
> > > java.net.ConnectException: Connection timed out
> > > at sun.nio.ch.Net.connect(Native Method)
> > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at
> > >
> > kafka.controller.RequestSendThread.connectToBroker(ControllerChannelMa
> > nager.scala:172)
> > > at
> > >
> > kafka.controller.RequestSendThread.<init>(ControllerChannelManager.sca
> > la:115)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.kafka$controller$ControllerC
> > hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > > at
> > >
> > kafka.controller.ControllerChannelManager.addBroker(ControllerChannelM
> > anager.scala:65)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.sca
> > la:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.sca
> > la:265)
> > > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> > dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> > kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildC
> > hange(ReplicaStateMachine.scala:255)
> > > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > > at
> > > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup
> > callback
> > > for 3 (kafka.controller.KafkaController)
> > > [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> > > Invoking state change to OnlineReplica for replicas
> > > (kafka.controller.ReplicaStateMachine)
> > > [2014-02-19 03:38:22,595] INFO
> > > [Controller-3-to-broker-3-send-thread],
> > > Starting (kafka.controller.RequestSendThread)
> > > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> > > Broker change listener fired for path /brokers/ids with children
> > > 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> > > Newly added brokers: 2,1, deleted brokers: , all live brokers: 3,2,1
> > > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > >
> > >
> > > -----Original Message-----
> > > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > > balasubramanian.jayaraman@autodesk.com]
> > > Sent: Thursday, February 06, 2014 5:40 PM
> > > To: users@kafka.apache.org
> > > Subject: RE: Reg Exception in Kafka
> > >
> > > After correcting the spelling, it is registered to correct public IP.
> > >
> > > Thanks
> > > Bala
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Thursday, 6 February, 2014 2:04 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > I think you mis-spelled the property name. It should be "
> > > advertised.host.name", instead of advertise.host.name
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > Thanks Joel.
> > > >
> > > > It seems the Broker is registered to the zookeeper with the IP
> > > > 10.199.31.87. The output pf the command is given below.
> > > >
> > > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> > > > /brokers/ids/1
> > > > Connecting to 0.0.0.0:2181
> > > >
> > > > WATCHER::
> > > >
> > > > WatchedEvent state:SyncConnected type:None path:null
> > > >
> > > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > > vers
> > > > ion":1,"port":9092}
> > > > cZxid = 0x2d
> > > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu Feb
> > > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength = 89
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > > Sent: Thursday, February 06, 2014 5:56 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Use zookeeper-shell script:
> > > >
> > > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > > /brokers/ids/<your broker id>
> > > >
> > > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > > Jayaraman
> > > > (Contingent) wrote:
> > > > > Where should I look for these information. From the logs, I
> > > > > could see
> > > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server
> > > > is started in port 9082 and bind to IP 10.x.x.x.
> > > > > If I don't give the host.name in server.properties, I get "
> > > > java.nio.channels.UnresolvedAddressException" and If I give the
> > > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > > It is the same behavior as in 0.8.0.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > It seems what's registered in ZK (10.199.31.87 <
> > > > http://10.199.31.87:9094/>)
> > > > > is still the local ip, not the public one. Could you check the
> > > > > broker registration in zk (
> > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+struc
> > > > ture
> > > > s+in+Zookeeper
> > > > )
> > > > > and see what's the host/port of the broker?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I downloaded from the trunk and set up the properties
> > > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > > >
> > > > > > Even after this change, I get the ConnectException. The
> > > > > > details logs
> > > > are
> > > > > > given below. Is there any workaround for this?
> > > > > >
> > > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > > 54.241.44.129:9094
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > > > metadata
> > > > for 1
> > > > > > topic(s) Set(mytopic)
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > > partition info
> > > > for
> > > > > > topic mytopic
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0]
> > > > > > has
> > > > leader 3
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > > registered for
> > > > > > topic: mytopic are 0
> > > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with
> > > > > > no compression to [mytopic,0] [DEBUG] [main 2014-02-04
> > > > > > 17:46:01,833] Producer sending messages with correlation id 2
> > > > > > for topics [mytopic,0] to broker 3 on
> > > > 10.199.31.87:9094
> > > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > > > > 10.199.31.87:9094 unsuccessful
> > > > > > java.net.ConnectException: Connection timed out: connect
> > > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > at
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:1
> > > 61)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > ncPr
> > > > oducer.scala:68)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > > 1.ap
> > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > > 1.ap
> > > > ply(SyncProducer.scala:102)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > scala:101)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la:1
> > > > 01)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la:1
> > > > 01)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > ultE
> > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > edDa
> > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > scal
> > > > a:80)
> > > > > > at
> > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > at
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:1
> > > 61)
> > > > > > at
> > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > 194)
> > > > > > at
> > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > at
> > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > faul
> > > > tEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandle
> > > > r.sc
> > > > ala:72)
> > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > at
> > > > > >
> > > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > > java:48)
> > > > > > at
> > > > > >
> > > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer
> > > > (Kaf
> > > > kaProducerTest.java:33)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > > Method)
> > > > > > at
> > > > > >
> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
> > > > pl.j
> > > > ava:57)
> > > > > > at
> > > > > >
> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
> > > > cess
> > > > orImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > > at
> > junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > > at
> > > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.r
> > > > un(J
> > > > Unit3TestReference.java:131)
> > > > > > at
> > > > > >
> > > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > > java:38)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > mote
> > > > TestRunner.java:467)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > > mote
> > > > TestRunner.java:683)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteT
> > > > estR
> > > > unner.java:390)
> > > > > > at
> > > > > >
> > > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remote
> > > > Test
> > > > Runner.java:197)
> > > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > > > > request with correlation id 2 to broker 3 with data for
> > > > > > partitions [mytopic,0]
> > > > > >
> > > > > >
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > > Could
> > > > some
> > > > > > EC2 users here help out?
> > > > > >
> > > > > > Also, we recently added
> > > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > > which allows one to use a different ip for binding and
> connecting.
> > > > > > You
> > > > can
> > > > > > see if this works for you. The patch is only in trunk though.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > > > (Contingent) <
> > > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > > Actually the public IP is not one of the IP configured in
> > > > > > > the Ethernet
> > > > interfaces.
> > > > > > > Only the Local IP is configured in eth0.
> > > > > > > Is there any solution to this?
> > > > > > >
> > > > > > > Ifconfig O/P:
> > > > > > >
> > > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > > Mask:255.255.255.192
> > > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > > RX packets:83186 errors:0 dropped:0 overruns:0
> frame:0
> > > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> > carrier:0
> > > > > > > collisions:0 txqueuelen:1000
> > > > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154
> > > > > > > (14.3
> > > MiB)
> > > > > > > Interrupt:25
> > > > > > >
> > > > > > > lo Link encap:Local Loopback
> > > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > > RX packets:1379711 errors:0 dropped:0 overruns:0
> > frame:0
> > > > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> > > carrier:0
> > > > > > > collisions:0 txqueuelen:0
> > > > > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > > > > (104.0
> > > > > > > MiB)
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Could it be a port conflict?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > Jun,
> > > > > > > >
> > > > > > > > Thanks for your help.
> > > > > > > > I get the following exception :
> > > > > > > > kafka.common.KafkaException: Socket server failed to bind
> > > > > > > > to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.sca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > >
> > > > > > > > The entire stack trace of the logs are placed below.
> > > > > > > >
> > > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > > overridden to
> > > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden
> > > > > > > > to
> > > > > > > > 9092
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > > overridden to
> > > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > log.cleanup.interval.mins
> > > > is
> > > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect
> > > > > > > > is overridden to
> > > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > > overridden
> > > > to 1
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > log.retention.hours is overridden to 168
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > num.network.threads is overridden to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> > > > overridden
> > > > > > > > to 2
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes
> > > > > > > > is overridden to
> > > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > > > Starting log flusher every 3000 ms with the following
> > > > > > > > overrides Map()
> > > > > > > > (kafka.log.LogManager)
> > > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > > KafkaServerStable
> > > > > > > > startup. Prepare to shutdown
> > > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > > kafka.common.KafkaException: Socket server failed to bind
> > > > > > > > to
> > > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 88)
> > > > > > > > at
> > kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > > at
> > > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > > at
> > kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.sca
> > > > la:3
> > > > 4)
> > > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.ja
> > > > va:1
> > > > 39)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :77)
> > > > > > > > at
> > > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > > :70)
> > > > > > > > at
> > > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > > 84)
> > > > > > > > ... 6 more
> > > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting
> > > > > > > > down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutting
> > > > > > > > down
> > > > > > > > (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker
> > > > > > > > 1],
> > > > Shutdown
> > > > > > > > completed (kafka.network.SocketServer)
> > > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> > > > completed
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting
> > > > > > > > down
> > > > > > > > (kafka.server.KafkaServer)
> > > > > > > >
> > > > > > > > Regards
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > You should should use the public IP for host.name. What's
> > > > > > > > the
> > > > error
> > > > > > > > you see during broker startup?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > I checked the faq. I did change the host.name in server
> > > > properties.
> > > > > > > > > After changing it I get ConnectException.
> > > > > > > > >
> > > > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > > > address
> > > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > > I set the host.name property to the local IP address
> > > > > > > > > which is 10.x.x.x. I think because of this there is a
> > > ConnectException.
> > > > > > > > > When I set the host.name to the public ip address
> > > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > > >
> > > > > > > > > What should be the IP address that is to be given in
> > > > > > host.nameproperty.
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > > > -----Original Message-----
> > > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > > To: users@kafka.apache.org
> > > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > > >
> > > > > > > > > Have you looked at
> > > > > > > > >
> > > > > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > > ?
> > > > > > > > >
> > > > > > > > > thanks,
> > > > > > > > >
> > > > > > > > > Jun
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > > Jayaraman
> > > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have a remote server (EC2) setup with Kafka cluster
> > setup.
> > > > > > > > > > There are
> > > > > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> > > > zookeeper
> > > > > > > > > > is running in the port 2181.
> > > > > > > > > > When I send message to the brokers from my PC, I get
> > > > > > > > > > an exception which is given below. I did a dump in the
> > > > > > > > > > remote server, the request is received in the remote
> server.
> > > > > > > > > > I am able to locally test the consumer/producer script
> > > > > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > > > > kindly help me in this
> > > > > > > > error?
> > > > > > > > > > Any help will be highly grateful.
> > > > > > > > > >
> > > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:90
> > > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class
> > > > > > > > > > is overridden to kafka.serializer.StringEncoder [
> > > > > > > > > > INFO] [main
> > > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> > > > overridden
> > > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > > > > 16:06:50,154]
> > > > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id
> > > > > > > > > > 0 for 1
> > > > > > > > > > topic(s)
> > > > > > > > > > Set(mytopic)
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket
> > > > > > > > > > with SO_TIMEOUT =
> > > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested
> > > > > > > > > > -1), SO_SNDBUF =
> > > > > > > > > > 102400 (requested 102400).
> > > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > > > 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > > 54.241.44.129:9093
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > > > fetched metadata for 1
> > > > > > > > > > topic(s) Set(mytopic)
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > > > > partition info for topic mytopic [DEBUG] [main
> > > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072] Broker
> > > > > > > > > > partitions registered for
> > > > > > > > > > topic: mytopic are 0
> > > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1
> > > > > > > > > > messages with no compression to [mytopic,0] [DEBUG]
> > > > > > > > > > [main
> > > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > > with correlation id 2 for topics [mytopic,0] to broker
> > > > > > > > > > 2 on
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > > connection to
> > > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > > at
> > > > > > > > >
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > > 640)
> > > > > > > > > > at
> > > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scal
> > > > > > > > a:57
> > > > > > > > )
> > > > > > > > > > at
> > > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > > > > nc
> > > > > > > > > Pr
> > > > > > > > > oducer.scala:68)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > > ap
> > > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > > scala:101)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > > la
> > > > > > > > :1
> > > > > > > > 01)
> > > > > > > > > > at
> > kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > > at
> > > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > > > > ul
> > > > > > > > > tE
> > > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > > ed
> > > > > > > > > Da
> > > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > > sc
> > > > > > > > > al
> > > > > > > > > a:80)
> > > > > > > > > > at
> > > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > >
> > > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala
> > > > :161
> > > > > > > )
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > > 194)
> > > > > > > > > > at
> > > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scal
> > > > > > > > a:39
> > > > > > > > )
> > > > > > > > > > at
> > > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > > > > fa
> > > > > > > > > ul
> > > > > > > > > tEventHandler.scala:100)
> > > > > > > > > > at
> > > > > > > > > >
> > > > > > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > > sc
> > > > > > > > > ala:72)
> > > > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > > at
> > > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > > at
> > > > > > > > > >
> > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > > > > at
> > > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.ja
> > > > > > > > > > va:2
> > > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to
> > > > > > > > > > send producer request with correlation id 2 to broker
> > > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > > >
> > > > > > > > > > Thanks
> > > > > > > > > > Bala
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
All the brokers reside in the same server and are listening on the port 10092,10093,10094. From the same machine I can connect to the zookeeper and the brokers. But When I tried to connect from an external machine (from Eclipse), I get an exception as communicated earlier. I was not able to connect to any of the brokers. I get the same exception while connecting to all the brokers.
Regards
Bala
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Thursday, 20 February, 2014 12:05 AM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
Can you make the connection from the controller host to the other broker?
Also, what's the # open file handlers on each broker? Do you see any "too many open file handler" error in the broker log?
Thanks,
Jun
On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> I don't think so. I am able to connect to the server using a putty. It
> is VM running in Amazon cloud.
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, 19 February, 2014 12:32 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Any issue with the network?
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman
> (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I just came from vacation.
> > When I tested it, I get a Connect Exception. The complete stack
> > trace is given below.
> >
> > Exception Stack Trace
> >
> > Java Thin Client:
> >
> > ava.net.ConnectException: Connection timed out: connect
> > at sun.nio.ch.Net.connect0(Native Method)
> > at sun.nio.ch.Net.connect(Net.java:465)
> > at sun.nio.ch.Net.connect(Net.java:457)
> > at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > at
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > at
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > at
> >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> oducer.scala:68)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > at
> > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > at
> >
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scal
> a:82)
> > at
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$s
> p(DefaultEventHandler.scala:67)
> > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > at
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> ala:67)
> > at kafka.producer.Producer.send(Producer.scala:76)
> > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > at
> >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> java:48)
> > at
> >
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> kaProducerTest.java:33)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at junit.framework.TestCase.runTest(TestCase.java:154)
> > at junit.framework.TestCase.runBare(TestCase.java:127)
> > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > at junit.framework.TestResult.run(TestResult.java:109)
> > at junit.framework.TestCase.run(TestCase.java:118)
> > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > at junit.framework.TestSuite.run(TestSuite.java:203)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> Unit3TestReference.java:131)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> java:38)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> TestRunner.java:467)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> TestRunner.java:683)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> unner.java:390)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> Runner.java:197)
> > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata with
> > correlation id 0 for topics [Set(mytopic)] from broker
> > [id:2,host:54.xx.xx.xx,port:10094] failed
> >
> > Kafka Server:
> > [2014-02-19 03:38:22,587] ERROR
> > [Controller-3-to-broker-3-send-thread],
> > Controller 3's connection to broker id:3,host:54.xx.xx.xx,port:10094
> > was unsuccessful (kafka.controller.RequestSendThread)
> > java.net.ConnectException: Connection timed out
> > at sun.nio.ch.Net.connect(Native Method)
> > at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > at
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > at
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelMa
> nager.scala:172)
> > at
> >
> kafka.controller.RequestSendThread.<init>(ControllerChannelManager.sca
> la:115)
> > at
> >
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerC
> hannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > at
> >
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelM
> anager.scala:65)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.sca
> la:265)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> dleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.sca
> la:265)
> > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> dleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$han
> dleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildC
> hange(ReplicaStateMachine.scala:255)
> > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > at
> > org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup
> callback
> > for 3 (kafka.controller.KafkaController)
> > [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> > Invoking state change to OnlineReplica for replicas
> > (kafka.controller.ReplicaStateMachine)
> > [2014-02-19 03:38:22,595] INFO
> > [Controller-3-to-broker-3-send-thread],
> > Starting (kafka.controller.RequestSendThread)
> > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> > Broker change listener fired for path /brokers/ids with children
> > 3,2,1
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> > Newly added brokers: 2,1, deleted brokers: , all live brokers: 3,2,1
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> >
> >
> > -----Original Message-----
> > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > balasubramanian.jayaraman@autodesk.com]
> > Sent: Thursday, February 06, 2014 5:40 PM
> > To: users@kafka.apache.org
> > Subject: RE: Reg Exception in Kafka
> >
> > After correcting the spelling, it is registered to correct public IP.
> >
> > Thanks
> > Bala
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, 6 February, 2014 2:04 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > I think you mis-spelled the property name. It should be "
> > advertised.host.name", instead of advertise.host.name
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > Thanks Joel.
> > >
> > > It seems the Broker is registered to the zookeeper with the IP
> > > 10.199.31.87. The output pf the command is given below.
> > >
> > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> > > /brokers/ids/1
> > > Connecting to 0.0.0.0:2181
> > >
> > > WATCHER::
> > >
> > > WatchedEvent state:SyncConnected type:None path:null
> > >
> > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","
> > > vers
> > > ion":1,"port":9092}
> > > cZxid = 0x2d
> > > ctime = Thu Feb 06 01:56:04 UTC 2014 mZxid = 0x2d mtime = Thu Feb
> > > 06 01:56:04 UTC 2014 pZxid = 0x2d cversion = 0 dataVersion = 0
> > > aclVersion = 0 ephemeralOwner = 0x14404e371520000 dataLength = 89
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > Sent: Thursday, February 06, 2014 5:56 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Use zookeeper-shell script:
> > >
> > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > /brokers/ids/<your broker id>
> > >
> > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian
> > > Jayaraman
> > > (Contingent) wrote:
> > > > Where should I look for these information. From the logs, I
> > > > could see
> > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server
> > > is started in port 9082 and bind to IP 10.x.x.x.
> > > > If I don't give the host.name in server.properties, I get "
> > > java.nio.channels.UnresolvedAddressException" and If I give the
> > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > It is the same behavior as in 0.8.0.
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > It seems what's registered in ZK (10.199.31.87 <
> > > http://10.199.31.87:9094/>)
> > > > is still the local ip, not the public one. Could you check the
> > > > broker registration in zk (
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+struc
> > > ture
> > > s+in+Zookeeper
> > > )
> > > > and see what's the host/port of the broker?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I downloaded from the trunk and set up the properties
> > > > > host.name= <local IP> advertise.host.name=<public IP>
> > > > >
> > > > > Even after this change, I get the ConnectException. The
> > > > > details logs
> > > are
> > > > > given below. Is there any workaround for this?
> > > > >
> > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > 54.241.44.129:9094
> > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > > metadata
> > > for 1
> > > > > topic(s) Set(mytopic)
> > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker
> > > > > partition info
> > > for
> > > > > topic mytopic
> > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0]
> > > > > has
> > > leader 3
> > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > registered for
> > > > > topic: mytopic are 0
> > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with
> > > > > no compression to [mytopic,0] [DEBUG] [main 2014-02-04
> > > > > 17:46:01,833] Producer sending messages with correlation id 2
> > > > > for topics [mytopic,0] to broker 3 on
> > > 10.199.31.87:9094
> > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > > > 10.199.31.87:9094 unsuccessful
> > > > > java.net.ConnectException: Connection timed out: connect
> > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > at
> > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > at
> > > > >
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:1
> > 61)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > ncPr
> > > oducer.scala:68)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > 1.ap
> > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > 1.ap
> > > ply(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$
> > > 1.ap
> > > ply(SyncProducer.scala:102)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > scala:101)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > la:1
> > > 01)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > la:1
> > > 01)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > ultE
> > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > edDa
> > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > edDa
> > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > at
> > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > scal
> > > a:80)
> > > > > at
> > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > scal
> > > a:80)
> > > > > at
> > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > at
> > > > >
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:1
> > 61)
> > > > > at
> > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > 194)
> > > > > at
> > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > at
> scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > faul
> > > tEventHandler.scala:100)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandle
> > > r.sc
> > > ala:72)
> > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > at
> > > > >
> > > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > java:48)
> > > > > at
> > > > >
> > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer
> > > (Kaf
> > > kaProducerTest.java:33)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > > > > at
> > > > >
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm
> > > pl.j
> > > ava:57)
> > > > > at
> > > > >
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAc
> > > cess
> > > orImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > at
> junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > at
> > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.r
> > > un(J
> > > Unit3TestReference.java:131)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > java:38)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > mote
> > > TestRunner.java:467)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Re
> > > mote
> > > TestRunner.java:683)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteT
> > > estR
> > > unner.java:390)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(Remote
> > > Test
> > > Runner.java:197)
> > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > > > request with correlation id 2 to broker 3 with data for
> > > > > partitions [mytopic,0]
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > Could
> > > some
> > > > > EC2 users here help out?
> > > > >
> > > > > Also, we recently added
> > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > which allows one to use a different ip for binding and connecting.
> > > > > You
> > > can
> > > > > see if this works for you. The patch is only in trunk though.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > > (Contingent) <
> > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > Actually the public IP is not one of the IP configured in
> > > > > > the Ethernet
> > > interfaces.
> > > > > > Only the Local IP is configured in eth0.
> > > > > > Is there any solution to this?
> > > > > >
> > > > > > Ifconfig O/P:
> > > > > >
> > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > Mask:255.255.255.192
> > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> carrier:0
> > > > > > collisions:0 txqueuelen:1000
> > > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154
> > > > > > (14.3
> > MiB)
> > > > > > Interrupt:25
> > > > > >
> > > > > > lo Link encap:Local Loopback
> > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > RX packets:1379711 errors:0 dropped:0 overruns:0
> frame:0
> > > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> > carrier:0
> > > > > > collisions:0 txqueuelen:0
> > > > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > > > (104.0
> > > > > > MiB)
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Could it be a port conflict?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for your help.
> > > > > > > I get the following exception :
> > > > > > > kafka.common.KafkaException: Socket server failed to bind
> > > > > > > to
> > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > 88)
> > > > > > > at
> kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > at
> > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > at
> kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > at
> > > > > > >
> > > > >
> > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.sca
> > > la:3
> > > 4)
> > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > at
> > > > > > >
> > > > >
> > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.ja
> > > va:1
> > > 39)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > :77)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > :70)
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > 84)
> > > > > > > ... 6 more
> > > > > > >
> > > > > > > The entire stack trace of the logs are placed below.
> > > > > > >
> > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > overridden to
> > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden
> > > > > > > to
> > > > > > > 9092
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> > > overridden
> > > > > > > to 2
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is
> > > > > > > overridden to
> > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > log.cleanup.interval.mins
> > > is
> > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,179] INFO Property
> > > > > > > log.flush.interval.ms is overridden to 1000
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect
> > > > > > > is overridden to
> > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > overridden
> > > to 1
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > log.retention.hours is overridden to 168
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > num.network.threads is overridden to 2
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> > > overridden
> > > > > > > to 2
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes
> > > > > > > is overridden to
> > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > > > (kafka.server.KafkaServer)
> > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > > Starting log flusher every 3000 ms with the following
> > > > > > > overrides Map()
> > > > > > > (kafka.log.LogManager)
> > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > KafkaServerStable
> > > > > > > startup. Prepare to shutdown
> > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > kafka.common.KafkaException: Socket server failed to bind
> > > > > > > to
> > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > 88)
> > > > > > > at
> kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > at
> > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > at
> kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > at
> > > > > > >
> > > > >
> > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.sca
> > > la:3
> > > 4)
> > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > at
> > > > > > >
> > > > >
> > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.ja
> > > va:1
> > > 39)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > :77)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java
> > > > > > :70)
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:1
> > > > > > 84)
> > > > > > > ... 6 more
> > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting
> > > > > > > down
> > > > > > > (kafka.server.KafkaServer)
> > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker
> > > > > > > 1],
> > > Shutting
> > > > > > > down
> > > > > > > (kafka.network.SocketServer)
> > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker
> > > > > > > 1],
> > > Shutdown
> > > > > > > completed (kafka.network.SocketServer)
> > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> > > completed
> > > > > > > (kafka.server.KafkaServer)
> > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting
> > > > > > > down
> > > > > > > (kafka.server.KafkaServer)
> > > > > > >
> > > > > > > Regards
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > You should should use the public IP for host.name. What's
> > > > > > > the
> > > error
> > > > > > > you see during broker startup?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > I checked the faq. I did change the host.name in server
> > > properties.
> > > > > > > > After changing it I get ConnectException.
> > > > > > > >
> > > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > > address
> > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > I set the host.name property to the local IP address
> > > > > > > > which is 10.x.x.x. I think because of this there is a
> > ConnectException.
> > > > > > > > When I set the host.name to the public ip address
> > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > >
> > > > > > > > What should be the IP address that is to be given in
> > > > > host.nameproperty.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > Have you looked at
> > > > > > > >
> > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > ?
> > > > > > > >
> > > > > > > > thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian
> > > > > > > > Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I have a remote server (EC2) setup with Kafka cluster
> setup.
> > > > > > > > > There are
> > > > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> > > zookeeper
> > > > > > > > > is running in the port 2181.
> > > > > > > > > When I send message to the brokers from my PC, I get
> > > > > > > > > an exception which is given below. I did a dump in the
> > > > > > > > > remote server, the request is received in the remote server.
> > > > > > > > > I am able to locally test the consumer/producer script
> > > > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > > > kindly help me in this
> > > > > > > error?
> > > > > > > > > Any help will be highly grateful.
> > > > > > > > >
> > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108]
> > > > > > > > > Property metadata.broker.list is overridden to
> > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:90
> > > > > > > > > 94 [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class
> > > > > > > > > is overridden to kafka.serializer.StringEncoder [
> > > > > > > > > INFO] [main
> > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> > > overridden
> > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > > > 16:06:50,154]
> > > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id
> > > > > > > > > 0 for 1
> > > > > > > > > topic(s)
> > > > > > > > > Set(mytopic)
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket
> > > > > > > > > with SO_TIMEOUT =
> > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested
> > > > > > > > > -1), SO_SNDBUF =
> > > > > > > > > 102400 (requested 102400).
> > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > > 54.241.44.129:9093for producing [ INFO] [main
> > > > > > > > > 2014-01-27 15:59:44,018] Disconnecting from
> > > > > > > > > 54.241.44.129:9093
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > > fetched metadata for 1
> > > > > > > > > topic(s) Set(mytopic)
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > > > partition info for topic mytopic [DEBUG] [main
> > > > > > > > > 2014-01-27 15:59:44,060] Partition [mytopic,0] has
> > > > > > > > > leader 2 [DEBUG] [main 2014-01-27 15:59:44,072] Broker
> > > > > > > > > partitions registered for
> > > > > > > > > topic: mytopic are 0
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1
> > > > > > > > > messages with no compression to [mytopic,0] [DEBUG]
> > > > > > > > > [main
> > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages
> > > > > > > > > with correlation id 2 for topics [mytopic,0] to broker
> > > > > > > > > 2 on
> > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer
> > > > > > > > > connection to
> > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > 640)
> > > > > > > > > at
> > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scal
> > > > > > > a:57
> > > > > > > )
> > > > > > > > > at
> > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala
> > > :161
> > > > > > )
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > > > nc
> > > > > > > > Pr
> > > > > > > > oducer.scala:68)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > ap
> > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > ap
> > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > ap
> > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > at
> kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > scala:101)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > la
> > > > > > > :1
> > > > > > > 01)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.sca
> > > la
> > > > > > > :1
> > > > > > > 01)
> > > > > > > > > at
> kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > at
> > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > > > ul
> > > > > > > > tE
> > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > ed
> > > > > > > > Da
> > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > ed
> > > > > > > > Da
> > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > sc
> > > > > > > > al
> > > > > > > > a:80)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > sc
> > > > > > > > al
> > > > > > > > a:80)
> > > > > > > > > at
> > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala
> > > :161
> > > > > > )
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > 194)
> > > > > > > > > at
> > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scal
> > > > > > > a:39
> > > > > > > )
> > > > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > > > fa
> > > > > > > > ul
> > > > > > > > tEventHandler.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > sc
> > > > > > > > ala:72)
> > > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > at
> > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > at
> > > > > > > > >
> > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > > > at
> > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.ja
> > > > > > > > > va:2
> > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to
> > > > > > > > > send producer request with correlation id 2 to broker
> > > > > > > > > 2 with data for partitions [mytopic,0]
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
>
Re: Reg Exception in Kafka
Posted by Jun Rao <ju...@gmail.com>.
Can you make the connection from the controller host to the other broker?
Also, what's the # open file handlers on each broker? Do you see any "too
many open file handler" error in the broker log?
Thanks,
Jun
On Wed, Feb 19, 2014 at 12:14 AM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> I don't think so. I am able to connect to the server using a putty. It is
> VM running in Amazon cloud.
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, 19 February, 2014 12:32 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Any issue with the network?
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I just came from vacation.
> > When I tested it, I get a Connect Exception. The complete stack trace is
> > given below.
> >
> > Exception Stack Trace
> >
> > Java Thin Client:
> >
> > ava.net.ConnectException: Connection timed out: connect
> > at sun.nio.ch.Net.connect0(Native Method)
> > at sun.nio.ch.Net.connect(Net.java:465)
> > at sun.nio.ch.Net.connect(Net.java:457)
> > at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > at
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > at
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > at
> >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> > at
> > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> > at
> >
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> > at
> >
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
> > at kafka.utils.Utils$.swallow(Utils.scala:187)
> > at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> > at kafka.utils.Utils$.swallowError(Utils.scala:46)
> > at
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
> > at kafka.producer.Producer.send(Producer.scala:76)
> > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > at
> >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> > at
> >
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at junit.framework.TestCase.runTest(TestCase.java:154)
> > at junit.framework.TestCase.runBare(TestCase.java:127)
> > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > at junit.framework.TestResult.run(TestResult.java:109)
> > at junit.framework.TestCase.run(TestCase.java:118)
> > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > at junit.framework.TestSuite.run(TestSuite.java:203)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> > at
> >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> > [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata with
> > correlation id 0 for topics [Set(mytopic)] from broker
> > [id:2,host:54.xx.xx.xx,port:10094] failed
> >
> > Kafka Server:
> > [2014-02-19 03:38:22,587] ERROR [Controller-3-to-broker-3-send-thread],
> > Controller 3's connection to broker id:3,host:54.xx.xx.xx,port:10094 was
> > unsuccessful (kafka.controller.RequestSendThread)
> > java.net.ConnectException: Connection timed out
> > at sun.nio.ch.Net.connect(Native Method)
> > at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> > at
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > at
> >
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:172)
> > at
> >
> kafka.controller.RequestSendThread.<init>(ControllerChannelManager.scala:115)
> > at
> >
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> > at
> >
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:65)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
> > at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at
> >
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:255)
> > at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> > at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> > [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup
> callback
> > for 3 (kafka.controller.KafkaController)
> > [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> > Invoking state change to OnlineReplica for replicas
> > (kafka.controller.ReplicaStateMachine)
> > [2014-02-19 03:38:22,595] INFO [Controller-3-to-broker-3-send-thread],
> > Starting (kafka.controller.RequestSendThread)
> > [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> > Broker change listener fired for path /brokers/ids with children 3,2,1
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> > [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> > Newly added brokers: 2,1, deleted brokers: , all live brokers: 3,2,1
> > (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> >
> >
> > -----Original Message-----
> > From: Balasubramanian Jayaraman (Contingent) [mailto:
> > balasubramanian.jayaraman@autodesk.com]
> > Sent: Thursday, February 06, 2014 5:40 PM
> > To: users@kafka.apache.org
> > Subject: RE: Reg Exception in Kafka
> >
> > After correcting the spelling, it is registered to correct public IP.
> >
> > Thanks
> > Bala
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, 6 February, 2014 2:04 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > I think you mis-spelled the property name. It should be "
> > advertised.host.name", instead of advertise.host.name
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman (Contingent) <
> > balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > Thanks Joel.
> > >
> > > It seems the Broker is registered to the zookeeper with the IP
> > > 10.199.31.87. The output pf the command is given below.
> > >
> > > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> > > /brokers/ids/1
> > > Connecting to 0.0.0.0:2181
> > >
> > > WATCHER::
> > >
> > > WatchedEvent state:SyncConnected type:None path:null
> > >
> > > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","vers
> > > ion":1,"port":9092}
> > > cZxid = 0x2d
> > > ctime = Thu Feb 06 01:56:04 UTC 2014
> > > mZxid = 0x2d
> > > mtime = Thu Feb 06 01:56:04 UTC 2014
> > > pZxid = 0x2d
> > > cversion = 0
> > > dataVersion = 0
> > > aclVersion = 0
> > > ephemeralOwner = 0x14404e371520000
> > > dataLength = 89
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > > Sent: Thursday, February 06, 2014 5:56 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Use zookeeper-shell script:
> > >
> > > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > > /brokers/ids/<your broker id>
> > >
> > > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman
> > > (Contingent) wrote:
> > > > Where should I look for these information. From the logs, I could
> > > > see
> > > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is
> > > started in port 9082 and bind to IP 10.x.x.x.
> > > > If I don't give the host.name in server.properties, I get "
> > > java.nio.channels.UnresolvedAddressException" and If I give the
> > host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > > It is the same behavior as in 0.8.0.
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > It seems what's registered in ZK (10.199.31.87 <
> > > http://10.199.31.87:9094/>)
> > > > is still the local ip, not the public one. Could you check the
> > > > broker registration in zk (
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structure
> > > s+in+Zookeeper
> > > )
> > > > and see what's the host/port of the broker?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I downloaded from the trunk and set up the properties host.name=
> > > > > <local IP> advertise.host.name=<public IP>
> > > > >
> > > > > Even after this change, I get the ConnectException. The details
> > > > > logs
> > > are
> > > > > given below. Is there any workaround for this?
> > > > >
> > > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > > 54.241.44.129:9094
> > > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > > metadata
> > > for 1
> > > > > topic(s) Set(mytopic)
> > > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition
> > > > > info
> > > for
> > > > > topic mytopic
> > > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has
> > > leader 3
> > > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > > registered for
> > > > > topic: mytopic are 0
> > > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > > > > compression to [mytopic,0] [DEBUG] [main 2014-02-04 17:46:01,833]
> > > > > Producer sending messages with correlation id 2 for topics
> > > > > [mytopic,0] to broker 3 on
> > > 10.199.31.87:9094
> > > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > > > 10.199.31.87:9094 unsuccessful
> > > > > java.net.ConnectException: Connection timed out: connect
> > > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > > at
> > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > at
> > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > at
> > > > >
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> > > oducer.scala:68)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > > ply(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > > ply(SyncProducer.scala:102)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > scala:101)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> > > 01)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> > > 01)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultE
> > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > at
> > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> > > a:80)
> > > > > at
> > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> > > a:80)
> > > > > at
> > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > at
> > > > >
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > > > > at
> > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > 194)
> > > > > at
> > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > at
> scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Defaul
> > > tEventHandler.scala:100)
> > > > > at
> > > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> > > ala:72)
> > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > at
> > > > >
> > > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > > java:48)
> > > > > at
> > > > >
> > > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> > > kaProducerTest.java:33)
> > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > > > > at
> > > > >
> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > > ava:57)
> > > > > at
> > > > >
> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > > orImpl.java:43)
> > > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > > at
> junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > > at
> > junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> > > Unit3TestReference.java:131)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > > java:38)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > > TestRunner.java:467)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > > TestRunner.java:683)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> > > unner.java:390)
> > > > > at
> > > > >
> > > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> > > Runner.java:197)
> > > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > > > request with correlation id 2 to broker 3 with data for partitions
> > > > > [mytopic,0]
> > > > >
> > > > >
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > > Could
> > > some
> > > > > EC2 users here help out?
> > > > >
> > > > > Also, we recently added
> > > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > > which allows one to use a different ip for binding and connecting.
> > > > > You
> > > can
> > > > > see if this works for you. The patch is only in trunk though.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > > (Contingent) <
> > > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > > Actually the public IP is not one of the IP configured in the
> > > > > > Ethernet
> > > interfaces.
> > > > > > Only the Local IP is configured in eth0.
> > > > > > Is there any solution to this?
> > > > > >
> > > > > > Ifconfig O/P:
> > > > > >
> > > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > > Mask:255.255.255.192
> > > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > > > > TX packets:91285 errors:0 dropped:0 overruns:0
> carrier:0
> > > > > > collisions:0 txqueuelen:1000
> > > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3
> > MiB)
> > > > > > Interrupt:25
> > > > > >
> > > > > > lo Link encap:Local Loopback
> > > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > > inet6 addr: ::1/128 Scope:Host
> > > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > > RX packets:1379711 errors:0 dropped:0 overruns:0
> frame:0
> > > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> > carrier:0
> > > > > > collisions:0 txqueuelen:0
> > > > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > > > (104.0
> > > > > > MiB)
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Could it be a port conflict?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > Jun,
> > > > > > >
> > > > > > > Thanks for your help.
> > > > > > > I get the following exception :
> > > > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > > > at
> kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > at
> > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > at
> kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > at
> > > > > > >
> > > > >
> > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> > > 4)
> > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > at
> > > > > > >
> > > > >
> > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> > > 39)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > > > ... 6 more
> > > > > > >
> > > > > > > The entire stack trace of the logs are placed below.
> > > > > > >
> > > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > > overridden to
> > > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to
> > > > > > > 9092
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> > > overridden
> > > > > > > to 2
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden
> > > > > > > to
> > > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > log.cleanup.interval.mins
> > > is
> > > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms
> > > > > > > is overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > > > > overridden to
> > > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > > overridden
> > > to 1
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> > > overridden
> > > > > > > to 2
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > > (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > > > > overridden to
> > > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > > > (kafka.server.KafkaServer)
> > > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > > Starting log flusher every 3000 ms with the following
> > > > > > > overrides Map()
> > > > > > > (kafka.log.LogManager)
> > > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > > KafkaServerStable
> > > > > > > startup. Prepare to shutdown
> > > > > > > (kafka.server.KafkaServerStartable)
> > > > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > > > at
> kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > > at
> > > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > > at
> kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > > at
> > > > > > >
> > > > >
> > > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> > > 4)
> > > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > > java.net.BindException: Cannot assign requested address
> > > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > > at
> > > > > > >
> > > > >
> > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> > > 39)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > > > at
> > > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > > > at
> > > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > > > ... 6 more
> > > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > > > > (kafka.server.KafkaServer)
> > > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1],
> > > Shutting
> > > > > > > down
> > > > > > > (kafka.network.SocketServer)
> > > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1],
> > > Shutdown
> > > > > > > completed (kafka.network.SocketServer)
> > > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > > (kafka.utils.KafkaScheduler)
> > > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> > > completed
> > > > > > > (kafka.server.KafkaServer)
> > > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > > > > (kafka.server.KafkaServer)
> > > > > > >
> > > > > > > Regards
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > You should should use the public IP for host.name. What's the
> > > error
> > > > > > > you see during broker startup?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > I checked the faq. I did change the host.name in server
> > > properties.
> > > > > > > > After changing it I get ConnectException.
> > > > > > > >
> > > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > > address
> > > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > > I set the host.name property to the local IP address which
> > > > > > > > is 10.x.x.x. I think because of this there is a
> > ConnectException.
> > > > > > > > When I set the host.name to the public ip address
> > > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > > >
> > > > > > > > What should be the IP address that is to be given in
> > > > > host.nameproperty.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Bala
> > > > > > > >
> > > > > > > > -----Original Message-----
> > > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > > To: users@kafka.apache.org
> > > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > > >
> > > > > > > > Have you looked at
> > > > > > > >
> > > > > > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > > ?
> > > > > > > >
> > > > > > > > thanks,
> > > > > > > >
> > > > > > > > Jun
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com>
> wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I have a remote server (EC2) setup with Kafka cluster
> setup.
> > > > > > > > > There are
> > > > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> > > zookeeper
> > > > > > > > > is running in the port 2181.
> > > > > > > > > When I send message to the brokers from my PC, I get an
> > > > > > > > > exception which is given below. I did a dump in the remote
> > > > > > > > > server, the request is received in the remote server.
> > > > > > > > > I am able to locally test the consumer/producer script
> > > > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > > > kindly help me in this
> > > > > > > error?
> > > > > > > > > Any help will be highly grateful.
> > > > > > > > >
> > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > metadata.broker.list is overridden to
> > > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> > > overridden
> > > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > > > 16:06:50,154]
> > > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0
> > > > > > > > > for 1
> > > > > > > > > topic(s)
> > > > > > > > > Set(mytopic)
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > > > > SO_TIMEOUT =
> > > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > > > > SO_SNDBUF =
> > > > > > > > > 102400 (requested 102400).
> > > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > > > > 15:59:44,018] Disconnecting from
> > > > > > > > > 54.241.44.129:9093
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > > fetched metadata for 1
> > > > > > > > > topic(s) Set(mytopic)
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > > > partition info for topic mytopic [DEBUG] [main 2014-01-27
> > > > > > > > > 15:59:44,060] Partition [mytopic,0] has leader 2 [DEBUG]
> > > > > > > > > [main 2014-01-27 15:59:44,072] Broker partitions
> > > > > > > > > registered for
> > > > > > > > > topic: mytopic are 0
> > > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages
> > > > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages with
> > > > > > > > > correlation id 2 for topics [mytopic,0] to broker 2 on
> > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection
> > > > > > > > > to
> > > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > > at
> > > > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > > 640)
> > > > > > > > > at
> > > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > > > )
> > > > > > > > > at
> > > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > > > > )
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > > > nc
> > > > > > > > Pr
> > > > > > > > oducer.scala:68)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > ap
> > > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > ap
> > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > > ap
> > > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > > at
> kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > > scala:101)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > > > :1
> > > > > > > 01)
> > > > > > > > > at
> > > > > > > > >
> > > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > > > :1
> > > > > > > 01)
> > > > > > > > > at
> kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > > at
> > > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > > > ul
> > > > > > > > tE
> > > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > ed
> > > > > > > > Da
> > > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > > ed
> > > > > > > > Da
> > > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > sc
> > > > > > > > al
> > > > > > > > a:80)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > > sc
> > > > > > > > al
> > > > > > > > a:80)
> > > > > > > > > at
> > > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > > at
> > > > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > > > > )
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > > 194)
> > > > > > > > > at
> > > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > > > )
> > > > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > > > fa
> > > > > > > > ul
> > > > > > > > tEventHandler.scala:100)
> > > > > > > > > at
> > > > > > > > >
> > > > > > > >
> > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > > sc
> > > > > > > > ala:72)
> > > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > > at
> > > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > > at
> > > > > > > > >
> > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > > > at
> > > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:2
> > > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to send
> > > > > > > > > producer request with correlation id 2 to broker 2 with
> > > > > > > > > data for partitions [mytopic,0]
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Bala
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> > >
> >
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
I don't think so. I am able to connect to the server using a putty. It is VM running in Amazon cloud.
Thanks
Bala
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Wednesday, 19 February, 2014 12:32 PM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
Any issue with the network?
Thanks,
Jun
On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> I just came from vacation.
> When I tested it, I get a Connect Exception. The complete stack trace is
> given below.
>
> Exception Stack Trace
>
> Java Thin Client:
>
> ava.net.ConnectException: Connection timed out: connect
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> at
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
> at kafka.utils.Utils$.swallow(Utils.scala:187)
> at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> at kafka.utils.Utils$.swallowError(Utils.scala:46)
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
> at kafka.producer.Producer.send(Producer.scala:76)
> at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> at
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> at
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:154)
> at junit.framework.TestCase.runBare(TestCase.java:127)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:118)
> at junit.framework.TestSuite.runTest(TestSuite.java:208)
> at junit.framework.TestSuite.run(TestSuite.java:203)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata with
> correlation id 0 for topics [Set(mytopic)] from broker
> [id:2,host:54.xx.xx.xx,port:10094] failed
>
> Kafka Server:
> [2014-02-19 03:38:22,587] ERROR [Controller-3-to-broker-3-send-thread],
> Controller 3's connection to broker id:3,host:54.xx.xx.xx,port:10094 was
> unsuccessful (kafka.controller.RequestSendThread)
> java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:172)
> at
> kafka.controller.RequestSendThread.<init>(ControllerChannelManager.scala:115)
> at
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> at
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:65)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:255)
> at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup callback
> for 3 (kafka.controller.KafkaController)
> [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> Invoking state change to OnlineReplica for replicas
> (kafka.controller.ReplicaStateMachine)
> [2014-02-19 03:38:22,595] INFO [Controller-3-to-broker-3-send-thread],
> Starting (kafka.controller.RequestSendThread)
> [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> Broker change listener fired for path /brokers/ids with children 3,2,1
> (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> Newly added brokers: 2,1, deleted brokers: , all live brokers: 3,2,1
> (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
>
>
> -----Original Message-----
> From: Balasubramanian Jayaraman (Contingent) [mailto:
> balasubramanian.jayaraman@autodesk.com]
> Sent: Thursday, February 06, 2014 5:40 PM
> To: users@kafka.apache.org
> Subject: RE: Reg Exception in Kafka
>
> After correcting the spelling, it is registered to correct public IP.
>
> Thanks
> Bala
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, 6 February, 2014 2:04 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> I think you mis-spelled the property name. It should be "
> advertised.host.name", instead of advertise.host.name
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > Thanks Joel.
> >
> > It seems the Broker is registered to the zookeeper with the IP
> > 10.199.31.87. The output pf the command is given below.
> >
> > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> > /brokers/ids/1
> > Connecting to 0.0.0.0:2181
> >
> > WATCHER::
> >
> > WatchedEvent state:SyncConnected type:None path:null
> >
> > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","vers
> > ion":1,"port":9092}
> > cZxid = 0x2d
> > ctime = Thu Feb 06 01:56:04 UTC 2014
> > mZxid = 0x2d
> > mtime = Thu Feb 06 01:56:04 UTC 2014
> > pZxid = 0x2d
> > cversion = 0
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x14404e371520000
> > dataLength = 89
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > Sent: Thursday, February 06, 2014 5:56 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Use zookeeper-shell script:
> >
> > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > /brokers/ids/<your broker id>
> >
> > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman
> > (Contingent) wrote:
> > > Where should I look for these information. From the logs, I could
> > > see
> > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is
> > started in port 9082 and bind to IP 10.x.x.x.
> > > If I don't give the host.name in server.properties, I get "
> > java.nio.channels.UnresolvedAddressException" and If I give the
> host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > It is the same behavior as in 0.8.0.
> > >
> > > Thanks
> > > Bala
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > It seems what's registered in ZK (10.199.31.87 <
> > http://10.199.31.87:9094/>)
> > > is still the local ip, not the public one. Could you check the
> > > broker registration in zk (
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structure
> > s+in+Zookeeper
> > )
> > > and see what's the host/port of the broker?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I downloaded from the trunk and set up the properties host.name=
> > > > <local IP> advertise.host.name=<public IP>
> > > >
> > > > Even after this change, I get the ConnectException. The details
> > > > logs
> > are
> > > > given below. Is there any workaround for this?
> > > >
> > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > 54.241.44.129:9094
> > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > metadata
> > for 1
> > > > topic(s) Set(mytopic)
> > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition
> > > > info
> > for
> > > > topic mytopic
> > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has
> > leader 3
> > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > registered for
> > > > topic: mytopic are 0
> > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > > > compression to [mytopic,0] [DEBUG] [main 2014-02-04 17:46:01,833]
> > > > Producer sending messages with correlation id 2 for topics
> > > > [mytopic,0] to broker 3 on
> > 10.199.31.87:9094
> > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > > 10.199.31.87:9094 unsuccessful
> > > > java.net.ConnectException: Connection timed out: connect
> > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at
> kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > at
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > > at
> > > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> > oducer.scala:68)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > ply$mcV$sp(SyncProducer.scala:102)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > ply(SyncProducer.scala:102)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > ply(SyncProducer.scala:102)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > scala:101)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> > 01)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> > 01)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultE
> > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> > ta$1.apply(DefaultEventHandler.scala:106)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> > ta$1.apply(DefaultEventHandler.scala:100)
> > > > at
> > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> > a:80)
> > > > at
> > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> > a:80)
> > > > at
> scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > at
> > > >
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > > > at
> > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > 194)
> > > > at
> > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Defaul
> > tEventHandler.scala:100)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> > ala:72)
> > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > at
> > > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > java:48)
> > > > at
> > > >
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> > kaProducerTest.java:33)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > at
> > > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:57)
> > > > at
> > > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > at
> junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> > Unit3TestReference.java:131)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > java:38)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:467)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:683)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> > unner.java:390)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> > Runner.java:197)
> > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > > request with correlation id 2 to broker 3 with data for partitions
> > > > [mytopic,0]
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > Could
> > some
> > > > EC2 users here help out?
> > > >
> > > > Also, we recently added
> > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > which allows one to use a different ip for binding and connecting.
> > > > You
> > can
> > > > see if this works for you. The patch is only in trunk though.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > (Contingent) <
> > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > Actually the public IP is not one of the IP configured in the
> > > > > Ethernet
> > interfaces.
> > > > > Only the Local IP is configured in eth0.
> > > > > Is there any solution to this?
> > > > >
> > > > > Ifconfig O/P:
> > > > >
> > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > Mask:255.255.255.192
> > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > > > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > > > > collisions:0 txqueuelen:1000
> > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3
> MiB)
> > > > > Interrupt:25
> > > > >
> > > > > lo Link encap:Local Loopback
> > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > inet6 addr: ::1/128 Scope:Host
> > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> carrier:0
> > > > > collisions:0 txqueuelen:0
> > > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > > (104.0
> > > > > MiB)
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Could it be a port conflict?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Thanks for your help.
> > > > > > I get the following exception :
> > > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > at
> > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > at
> > > > > >
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> > 4)
> > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > java.net.BindException: Cannot assign requested address
> > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > at
> > > > > >
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> > 39)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > > ... 6 more
> > > > > >
> > > > > > The entire stack trace of the logs are placed below.
> > > > > >
> > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > overridden to
> > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to
> > > > > > 9092
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> > overridden
> > > > > > to 2
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden
> > > > > > to
> > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > log.cleanup.interval.mins
> > is
> > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms
> > > > > > is overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > > > overridden to
> > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > overridden
> > to 1
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> > overridden
> > > > > > to 2
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > > > overridden to
> > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > > (kafka.server.KafkaServer)
> > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > Starting log flusher every 3000 ms with the following
> > > > > > overrides Map()
> > > > > > (kafka.log.LogManager)
> > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > KafkaServerStable
> > > > > > startup. Prepare to shutdown
> > > > > > (kafka.server.KafkaServerStartable)
> > > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > at
> > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > at
> > > > > >
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> > 4)
> > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > java.net.BindException: Cannot assign requested address
> > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > at
> > > > > >
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> > 39)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > > ... 6 more
> > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > > > (kafka.server.KafkaServer)
> > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1],
> > Shutting
> > > > > > down
> > > > > > (kafka.network.SocketServer)
> > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1],
> > Shutdown
> > > > > > completed (kafka.network.SocketServer)
> > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > (kafka.utils.KafkaScheduler)
> > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> > completed
> > > > > > (kafka.server.KafkaServer)
> > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > > > (kafka.server.KafkaServer)
> > > > > >
> > > > > > Regards
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > You should should use the public IP for host.name. What's the
> > error
> > > > > > you see during broker startup?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I checked the faq. I did change the host.name in server
> > properties.
> > > > > > > After changing it I get ConnectException.
> > > > > > >
> > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > address
> > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > I set the host.name property to the local IP address which
> > > > > > > is 10.x.x.x. I think because of this there is a
> ConnectException.
> > > > > > > When I set the host.name to the public ip address
> > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > >
> > > > > > > What should be the IP address that is to be given in
> > > > host.nameproperty.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Have you looked at
> > > > > > >
> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > ?
> > > > > > >
> > > > > > > thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > > > > There are
> > > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> > zookeeper
> > > > > > > > is running in the port 2181.
> > > > > > > > When I send message to the brokers from my PC, I get an
> > > > > > > > exception which is given below. I did a dump in the remote
> > > > > > > > server, the request is received in the remote server.
> > > > > > > > I am able to locally test the consumer/producer script
> > > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > > kindly help me in this
> > > > > > error?
> > > > > > > > Any help will be highly grateful.
> > > > > > > >
> > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > metadata.broker.list is overridden to
> > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> > overridden
> > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > > 16:06:50,154]
> > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0
> > > > > > > > for 1
> > > > > > > > topic(s)
> > > > > > > > Set(mytopic)
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > > > SO_TIMEOUT =
> > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > > > SO_SNDBUF =
> > > > > > > > 102400 (requested 102400).
> > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > > > 15:59:44,018] Disconnecting from
> > > > > > > > 54.241.44.129:9093
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > fetched metadata for 1
> > > > > > > > topic(s) Set(mytopic)
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > > partition info for topic mytopic [DEBUG] [main 2014-01-27
> > > > > > > > 15:59:44,060] Partition [mytopic,0] has leader 2 [DEBUG]
> > > > > > > > [main 2014-01-27 15:59:44,072] Broker partitions
> > > > > > > > registered for
> > > > > > > > topic: mytopic are 0
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages
> > > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages with
> > > > > > > > correlation id 2 for topics [mytopic,0] to broker 2 on
> > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection
> > > > > > > > to
> > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > at
> > > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > 640)
> > > > > > > > at
> > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > > )
> > > > > > > > at
> > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > at
> > > > > > > >
> > > > >
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > > > )
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > > nc
> > > > > > > Pr
> > > > > > > oducer.scala:68)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > ap
> > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > ap
> > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > ap
> > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > scala:101)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > > :1
> > > > > > 01)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > > :1
> > > > > > 01)
> > > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > at
> > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > > ul
> > > > > > > tE
> > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > ed
> > > > > > > Da
> > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > ed
> > > > > > > Da
> > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > sc
> > > > > > > al
> > > > > > > a:80)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > sc
> > > > > > > al
> > > > > > > a:80)
> > > > > > > > at
> > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > at
> > > > > > > >
> > > > >
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > > > )
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > 194)
> > > > > > > > at
> > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > > )
> > > > > > > > at
> > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > > fa
> > > > > > > ul
> > > > > > > tEventHandler.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > sc
> > > > > > > ala:72)
> > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > at
> > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > at
> > > > > > > >
> > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > > at
> > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:2
> > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to send
> > > > > > > > producer request with correlation id 2 to broker 2 with
> > > > > > > > data for partitions [mytopic,0]
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Bala
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
>
Re: Reg Exception in Kafka
Posted by Jun Rao <ju...@gmail.com>.
Any issue with the network?
Thanks,
Jun
On Tue, Feb 18, 2014 at 7:48 PM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> I just came from vacation.
> When I tested it, I get a Connect Exception. The complete stack trace is
> given below.
>
> Exception Stack Trace
>
> Java Thin Client:
>
> ava.net.ConnectException: Connection timed out: connect
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
> at
> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
> at
> kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
> at kafka.utils.Utils$.swallow(Utils.scala:187)
> at kafka.utils.Logging$class.swallowError(Logging.scala:105)
> at kafka.utils.Utils$.swallowError(Utils.scala:46)
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
> at kafka.producer.Producer.send(Producer.scala:76)
> at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> at
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> at
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:154)
> at junit.framework.TestCase.runBare(TestCase.java:127)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:118)
> at junit.framework.TestSuite.runTest(TestSuite.java:208)
> at junit.framework.TestSuite.run(TestSuite.java:203)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> [ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata with
> correlation id 0 for topics [Set(mytopic)] from broker
> [id:2,host:54.xx.xx.xx,port:10094] failed
>
> Kafka Server:
> [2014-02-19 03:38:22,587] ERROR [Controller-3-to-broker-3-send-thread],
> Controller 3's connection to broker id:3,host:54.xx.xx.xx,port:10094 was
> unsuccessful (kafka.controller.RequestSendThread)
> java.net.ConnectException: Connection timed out
> at sun.nio.ch.Net.connect(Native Method)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at
> kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:172)
> at
> kafka.controller.RequestSendThread.<init>(ControllerChannelManager.scala:115)
> at
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:84)
> at
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:65)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
> at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:255)
> at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> [2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup callback
> for 3 (kafka.controller.KafkaController)
> [2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]:
> Invoking state change to OnlineReplica for replicas
> (kafka.controller.ReplicaStateMachine)
> [2014-02-19 03:38:22,595] INFO [Controller-3-to-broker-3-send-thread],
> Starting (kafka.controller.RequestSendThread)
> [2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]:
> Broker change listener fired for path /brokers/ids with children 3,2,1
> (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> [2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]:
> Newly added brokers: 2,1, deleted brokers: , all live brokers: 3,2,1
> (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
>
>
> -----Original Message-----
> From: Balasubramanian Jayaraman (Contingent) [mailto:
> balasubramanian.jayaraman@autodesk.com]
> Sent: Thursday, February 06, 2014 5:40 PM
> To: users@kafka.apache.org
> Subject: RE: Reg Exception in Kafka
>
> After correcting the spelling, it is registered to correct public IP.
>
> Thanks
> Bala
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, 6 February, 2014 2:04 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> I think you mis-spelled the property name. It should be "
> advertised.host.name", instead of advertise.host.name
>
> Thanks,
>
> Jun
>
>
> On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > Thanks Joel.
> >
> > It seems the Broker is registered to the zookeeper with the IP
> > 10.199.31.87. The output pf the command is given below.
> >
> > [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> > /brokers/ids/1
> > Connecting to 0.0.0.0:2181
> >
> > WATCHER::
> >
> > WatchedEvent state:SyncConnected type:None path:null
> >
> > {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","vers
> > ion":1,"port":9092}
> > cZxid = 0x2d
> > ctime = Thu Feb 06 01:56:04 UTC 2014
> > mZxid = 0x2d
> > mtime = Thu Feb 06 01:56:04 UTC 2014
> > pZxid = 0x2d
> > cversion = 0
> > dataVersion = 0
> > aclVersion = 0
> > ephemeralOwner = 0x14404e371520000
> > dataLength = 89
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> > Sent: Thursday, February 06, 2014 5:56 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Use zookeeper-shell script:
> >
> > ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> > /brokers/ids/<your broker id>
> >
> > On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman
> > (Contingent) wrote:
> > > Where should I look for these information. From the logs, I could
> > > see
> > ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is
> > started in port 9082 and bind to IP 10.x.x.x.
> > > If I don't give the host.name in server.properties, I get "
> > java.nio.channels.UnresolvedAddressException" and If I give the
> host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > > It is the same behavior as in 0.8.0.
> > >
> > > Thanks
> > > Bala
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, February 05, 2014 12:21 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > It seems what's registered in ZK (10.199.31.87 <
> > http://10.199.31.87:9094/>)
> > > is still the local ip, not the public one. Could you check the
> > > broker registration in zk (
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structure
> > s+in+Zookeeper
> > )
> > > and see what's the host/port of the broker?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I downloaded from the trunk and set up the properties host.name=
> > > > <local IP> advertise.host.name=<public IP>
> > > >
> > > > Even after this change, I get the ConnectException. The details
> > > > logs
> > are
> > > > given below. Is there any workaround for this?
> > > >
> > > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > > 54.241.44.129:9094
> > > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > > metadata
> > for 1
> > > > topic(s) Set(mytopic)
> > > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition
> > > > info
> > for
> > > > topic mytopic
> > > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has
> > leader 3
> > > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > > registered for
> > > > topic: mytopic are 0
> > > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > > > compression to [mytopic,0] [DEBUG] [main 2014-02-04 17:46:01,833]
> > > > Producer sending messages with correlation id 2 for topics
> > > > [mytopic,0] to broker 3 on
> > 10.199.31.87:9094
> > > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > > 10.199.31.87:9094 unsuccessful
> > > > java.net.ConnectException: Connection timed out: connect
> > > > at sun.nio.ch.Net.connect0(Native Method)
> > > > at sun.nio.ch.Net.connect(Net.java:465)
> > > > at sun.nio.ch.Net.connect(Net.java:457)
> > > > at
> > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > > at
> > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > at
> kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > at
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > > at
> > > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> > oducer.scala:68)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > ply$mcV$sp(SyncProducer.scala:102)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > ply(SyncProducer.scala:102)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> > ply(SyncProducer.scala:102)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > scala:101)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> > 01)
> > > > at
> > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> > 01)
> > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultE
> > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> > ta$1.apply(DefaultEventHandler.scala:106)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> > ta$1.apply(DefaultEventHandler.scala:100)
> > > > at
> > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> > a:80)
> > > > at
> > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> > a:80)
> > > > at
> scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > at
> > > >
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > > > at
> > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > 194)
> > > > at
> > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Defaul
> > tEventHandler.scala:100)
> > > > at
> > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> > ala:72)
> > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > at
> > > >
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> > java:48)
> > > > at
> > > >
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> > kaProducerTest.java:33)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > at
> > > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> > ava:57)
> > > > at
> > > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> > orImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > > at
> junit.framework.TestResult.runProtected(TestResult.java:124)
> > > > at junit.framework.TestResult.run(TestResult.java:109)
> > > > at junit.framework.TestCase.run(TestCase.java:118)
> > > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> > Unit3TestReference.java:131)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> > java:38)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:467)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> > TestRunner.java:683)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> > unner.java:390)
> > > > at
> > > >
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> > Runner.java:197)
> > > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > > request with correlation id 2 to broker 3 with data for partitions
> > > > [mytopic,0]
> > > >
> > > >
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > > Could
> > some
> > > > EC2 users here help out?
> > > >
> > > > Also, we recently added
> > https://issues.apache.org/jira/browse/KAFKA-1092,
> > > > which allows one to use a different ip for binding and connecting.
> > > > You
> > can
> > > > see if this works for you. The patch is only in trunk though.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> > (Contingent) <
> > > > balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I don't think so. I forgot to include the ifconfig output.
> > > > > Actually the public IP is not one of the IP configured in the
> > > > > Ethernet
> > interfaces.
> > > > > Only the Local IP is configured in eth0.
> > > > > Is there any solution to this?
> > > > >
> > > > > Ifconfig O/P:
> > > > >
> > > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > > inet addr:10.X.X.X Bcast:10.199.31.127
> > Mask:255.255.255.192
> > > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > > > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > > > > collisions:0 txqueuelen:1000
> > > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3
> MiB)
> > > > > Interrupt:25
> > > > >
> > > > > lo Link encap:Local Loopback
> > > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > > inet6 addr: ::1/128 Scope:Host
> > > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > > > > TX packets:1379711 errors:0 dropped:0 overruns:0
> carrier:0
> > > > > collisions:0 txqueuelen:0
> > > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > > (104.0
> > > > > MiB)
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Could it be a port conflict?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > Jun,
> > > > > >
> > > > > > Thanks for your help.
> > > > > > I get the following exception :
> > > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > at
> > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > at
> > > > > >
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> > 4)
> > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > java.net.BindException: Cannot assign requested address
> > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > at
> > > > > >
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> > 39)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > > ... 6 more
> > > > > >
> > > > > > The entire stack trace of the logs are placed below.
> > > > > >
> > > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > > overridden to
> > > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to
> > > > > > 9092
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > > socket.request.max.bytes is overridden to 104857600
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> > overridden
> > > > > > to 2
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden
> > > > > > to
> > > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > log.cleanup.interval.mins
> > is
> > > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms
> > > > > > is overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > > > overridden to
> > > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > > overridden
> > to 1
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> > overridden
> > > > > > to 2
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > > log.flush.interval.messages is overridden to 10000
> > > > > > (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > > > overridden to
> > > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > > (kafka.server.KafkaServer)
> > > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > > Starting log flusher every 3000 ms with the following
> > > > > > overrides Map()
> > > > > > (kafka.log.LogManager)
> > > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> > KafkaServerStable
> > > > > > startup. Prepare to shutdown
> > > > > > (kafka.server.KafkaServerStartable)
> > > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > > at
> > kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > > at
> > > > > >
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> > 4)
> > > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > > java.net.BindException: Cannot assign requested address
> > > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > > at
> > > > > >
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> > 39)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > > at
> > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > > at
> > > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > > ... 6 more
> > > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > > > (kafka.server.KafkaServer)
> > > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1],
> > Shutting
> > > > > > down
> > > > > > (kafka.network.SocketServer)
> > > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1],
> > Shutdown
> > > > > > completed (kafka.network.SocketServer)
> > > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > > (kafka.utils.KafkaScheduler)
> > > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> > completed
> > > > > > (kafka.server.KafkaServer)
> > > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > > > (kafka.server.KafkaServer)
> > > > > >
> > > > > > Regards
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > You should should use the public IP for host.name. What's the
> > error
> > > > > > you see during broker startup?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > I checked the faq. I did change the host.name in server
> > properties.
> > > > > > > After changing it I get ConnectException.
> > > > > > >
> > > > > > > The problem here is in EC2 have a different public IP
> > > > > > > address
> > > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > > I set the host.name property to the local IP address which
> > > > > > > is 10.x.x.x. I think because of this there is a
> ConnectException.
> > > > > > > When I set the host.name to the public ip address
> > > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > > >
> > > > > > > What should be the IP address that is to be given in
> > > > host.nameproperty.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > > > -----Original Message-----
> > > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > > To: users@kafka.apache.org
> > > > > > > Subject: Re: Reg Exception in Kafka
> > > > > > >
> > > > > > > Have you looked at
> > > > > > >
> > > > > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > > ?
> > > > > > >
> > > > > > > thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > > > > There are
> > > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> > zookeeper
> > > > > > > > is running in the port 2181.
> > > > > > > > When I send message to the brokers from my PC, I get an
> > > > > > > > exception which is given below. I did a dump in the remote
> > > > > > > > server, the request is received in the remote server.
> > > > > > > > I am able to locally test the consumer/producer script
> > > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > > kindly help me in this
> > > > > > error?
> > > > > > > > Any help will be highly grateful.
> > > > > > > >
> > > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > metadata.broker.list is overridden to
> > > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> > overridden
> > > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > > 16:06:50,154]
> > > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0
> > > > > > > > for 1
> > > > > > > > topic(s)
> > > > > > > > Set(mytopic)
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > > > SO_TIMEOUT =
> > > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > > > SO_SNDBUF =
> > > > > > > > 102400 (requested 102400).
> > > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > > > 15:59:44,018] Disconnecting from
> > > > > > > > 54.241.44.129:9093
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > > fetched metadata for 1
> > > > > > > > topic(s) Set(mytopic)
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > > partition info for topic mytopic [DEBUG] [main 2014-01-27
> > > > > > > > 15:59:44,060] Partition [mytopic,0] has leader 2 [DEBUG]
> > > > > > > > [main 2014-01-27 15:59:44,072] Broker partitions
> > > > > > > > registered for
> > > > > > > > topic: mytopic are 0
> > > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages
> > > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > > 2014-01-27 15:59:44,109] Producer sending messages with
> > > > > > > > correlation id 2 for topics [mytopic,0] to broker 2 on
> > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection
> > > > > > > > to
> > > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > > at
> > > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > > 640)
> > > > > > > > at
> > > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > > )
> > > > > > > > at
> > > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > > at
> > > > > > > >
> > > > >
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > > > )
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > > nc
> > > > > > > Pr
> > > > > > > oducer.scala:68)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > ap
> > > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > ap
> > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > > ap
> > > > > > > ply(SyncProducer.scala:102)
> > > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > > scala:101)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > > :1
> > > > > > 01)
> > > > > > > > at
> > > > > > > >
> > > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > > :1
> > > > > > 01)
> > > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > > at
> > kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > > ul
> > > > > > > tE
> > > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > ed
> > > > > > > Da
> > > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > > ed
> > > > > > > Da
> > > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > sc
> > > > > > > al
> > > > > > > a:80)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > > sc
> > > > > > > al
> > > > > > > a:80)
> > > > > > > > at
> > > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > > at
> > > > > > > >
> > > > >
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > > > )
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > > 194)
> > > > > > > > at
> > > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > > )
> > > > > > > > at
> > > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > > fa
> > > > > > > ul
> > > > > > > tEventHandler.scala:100)
> > > > > > > > at
> > > > > > > >
> > > > > > >
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > > sc
> > > > > > > ala:72)
> > > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > > at
> > kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > > at
> > > > > > > >
> > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > > at
> > > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:2
> > > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to send
> > > > > > > > producer request with correlation id 2 to broker 2 with
> > > > > > > > data for partitions [mytopic,0]
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Bala
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> >
> >
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
I just came from vacation.
When I tested it, I get a Connect Exception. The complete stack trace is given below.
Exception Stack Trace
Java Thin Client:
ava.net.ConnectException: Connection timed out: connect
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:465)
at sun.nio.ch.Net.connect(Net.java:457)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
at kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
at kafka.producer.SyncProducer.send(SyncProducer.scala:112)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:53)
at kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82)
at kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67)
at kafka.utils.Utils$.swallow(Utils.scala:187)
at kafka.utils.Logging$class.swallowError(Logging.scala:105)
at kafka.utils.Utils$.swallowError(Utils.scala:46)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:33)
at com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
at com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:154)
at junit.framework.TestCase.runBare(TestCase.java:127)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:118)
at junit.framework.TestSuite.runTest(TestSuite.java:208)
at junit.framework.TestSuite.run(TestSuite.java:203)
at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
[ WARN] [main 2014-02-19 11:38:04,594] Fetching topic metadata with correlation id 0 for topics [Set(mytopic)] from broker [id:2,host:54.xx.xx.xx,port:10094] failed
Kafka Server:
[2014-02-19 03:38:22,587] ERROR [Controller-3-to-broker-3-send-thread], Controller 3's connection to broker id:3,host:54.xx.xx.xx,port:10094 was unsuccessful (kafka.controller.RequestSendThread)
java.net.ConnectException: Connection timed out
at sun.nio.ch.Net.connect(Native Method)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:530)
at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
at kafka.controller.RequestSendThread.connectToBroker(ControllerChannelManager.scala:172)
at kafka.controller.RequestSendThread.<init>(ControllerChannelManager.scala:115)
at kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:84)
at kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:65)
at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$6.apply(ReplicaStateMachine.scala:265)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:265)
at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
at kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:256)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:255)
at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
[2014-02-19 03:38:22,590] INFO [Controller 3]: New broker startup callback for 3 (kafka.controller.KafkaController)
[2014-02-19 03:38:22,592] INFO [Replica state machine on controller 3]: Invoking state change to OnlineReplica for replicas (kafka.controller.ReplicaStateMachine)
[2014-02-19 03:38:22,595] INFO [Controller-3-to-broker-3-send-thread], Starting (kafka.controller.RequestSendThread)
[2014-02-19 03:38:22,611] INFO [BrokerChangeListener on Controller 3]: Broker change listener fired for path /brokers/ids with children 3,2,1 (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
[2014-02-19 03:38:22,834] INFO [BrokerChangeListener on Controller 3]: Newly added brokers: 2,1, deleted brokers: , all live brokers: 3,2,1 (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
-----Original Message-----
From: Balasubramanian Jayaraman (Contingent) [mailto:balasubramanian.jayaraman@autodesk.com]
Sent: Thursday, February 06, 2014 5:40 PM
To: users@kafka.apache.org
Subject: RE: Reg Exception in Kafka
After correcting the spelling, it is registered to correct public IP.
Thanks
Bala
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Thursday, 6 February, 2014 2:04 PM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
I think you mis-spelled the property name. It should be "
advertised.host.name", instead of advertise.host.name
Thanks,
Jun
On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> Thanks Joel.
>
> It seems the Broker is registered to the zookeeper with the IP
> 10.199.31.87. The output pf the command is given below.
>
> [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> /brokers/ids/1
> Connecting to 0.0.0.0:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
>
> {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","vers
> ion":1,"port":9092}
> cZxid = 0x2d
> ctime = Thu Feb 06 01:56:04 UTC 2014
> mZxid = 0x2d
> mtime = Thu Feb 06 01:56:04 UTC 2014
> pZxid = 0x2d
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x14404e371520000
> dataLength = 89
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> Sent: Thursday, February 06, 2014 5:56 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Use zookeeper-shell script:
>
> ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> /brokers/ids/<your broker id>
>
> On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman
> (Contingent) wrote:
> > Where should I look for these information. From the logs, I could
> > see
> ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is
> started in port 9082 and bind to IP 10.x.x.x.
> > If I don't give the host.name in server.properties, I get "
> java.nio.channels.UnresolvedAddressException" and If I give the host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > It is the same behavior as in 0.8.0.
> >
> > Thanks
> > Bala
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, February 05, 2014 12:21 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > It seems what's registered in ZK (10.199.31.87 <
> http://10.199.31.87:9094/>)
> > is still the local ip, not the public one. Could you check the
> > broker registration in zk (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structure
> s+in+Zookeeper
> )
> > and see what's the host/port of the broker?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I downloaded from the trunk and set up the properties host.name=
> > > <local IP> advertise.host.name=<public IP>
> > >
> > > Even after this change, I get the ConnectException. The details
> > > logs
> are
> > > given below. Is there any workaround for this?
> > >
> > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > 54.241.44.129:9094
> > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > metadata
> for 1
> > > topic(s) Set(mytopic)
> > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition
> > > info
> for
> > > topic mytopic
> > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has
> leader 3
> > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > registered for
> > > topic: mytopic are 0
> > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > > compression to [mytopic,0] [DEBUG] [main 2014-02-04 17:46:01,833]
> > > Producer sending messages with correlation id 2 for topics
> > > [mytopic,0] to broker 3 on
> 10.199.31.87:9094
> > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > 10.199.31.87:9094 unsuccessful
> > > java.net.ConnectException: Connection timed out: connect
> > > at sun.nio.ch.Net.connect0(Native Method)
> > > at sun.nio.ch.Net.connect(Net.java:465)
> > > at sun.nio.ch.Net.connect(Net.java:457)
> > > at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > at
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > at
> > >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> oducer.scala:68)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> ply$mcV$sp(SyncProducer.scala:102)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> ply(SyncProducer.scala:102)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> ply(SyncProducer.scala:102)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> scala:101)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> 01)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> 01)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultE
> ventHandler$$send(DefaultEventHandler.scala:254)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> ta$1.apply(DefaultEventHandler.scala:106)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> ta$1.apply(DefaultEventHandler.scala:100)
> > > at
> > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> a:80)
> > > at
> > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> a:80)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > at
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > > at
> > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> 194)
> > > at
> scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Defaul
> tEventHandler.scala:100)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> ala:72)
> > > at kafka.producer.Producer.send(Producer.scala:76)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at
> > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> java:48)
> > > at
> > >
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> kaProducerTest.java:33)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> > > at
> > >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > > at junit.framework.TestResult.run(TestResult.java:109)
> > > at junit.framework.TestCase.run(TestCase.java:118)
> > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> Unit3TestReference.java:131)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> java:38)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> TestRunner.java:467)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> TestRunner.java:683)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> unner.java:390)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> Runner.java:197)
> > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > request with correlation id 2 to broker 3 with data for partitions
> > > [mytopic,0]
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > Could
> some
> > > EC2 users here help out?
> > >
> > > Also, we recently added
> https://issues.apache.org/jira/browse/KAFKA-1092,
> > > which allows one to use a different ip for binding and connecting.
> > > You
> can
> > > see if this works for you. The patch is only in trunk though.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> (Contingent) <
> > > balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I don't think so. I forgot to include the ifconfig output.
> > > > Actually the public IP is not one of the IP configured in the
> > > > Ethernet
> interfaces.
> > > > Only the Local IP is configured in eth0.
> > > > Is there any solution to this?
> > > >
> > > > Ifconfig O/P:
> > > >
> > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > inet addr:10.X.X.X Bcast:10.199.31.127
> Mask:255.255.255.192
> > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > > > collisions:0 txqueuelen:1000
> > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> > > > Interrupt:25
> > > >
> > > > lo Link encap:Local Loopback
> > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > inet6 addr: ::1/128 Scope:Host
> > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > > > TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> > > > collisions:0 txqueuelen:0
> > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > (104.0
> > > > MiB)
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Could it be a port conflict?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thanks for your help.
> > > > > I get the following exception :
> > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > at
> kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > at
> > > > >
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> 4)
> > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > java.net.BindException: Cannot assign requested address
> > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > at
> > > > >
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> 39)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > ... 6 more
> > > > >
> > > > > The entire stack trace of the logs are placed below.
> > > > >
> > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > overridden to
> > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to
> > > > > 9092
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > socket.request.max.bytes is overridden to 104857600
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> overridden
> > > > > to 2
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden
> > > > > to
> > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > log.cleanup.interval.mins
> is
> > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms
> > > > > is overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > > overridden to
> > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > overridden
> to 1
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> overridden
> > > > > to 2
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > log.flush.interval.messages is overridden to 10000
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > > overridden to
> > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > Starting log flusher every 3000 ms with the following
> > > > > overrides Map()
> > > > > (kafka.log.LogManager)
> > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> KafkaServerStable
> > > > > startup. Prepare to shutdown
> > > > > (kafka.server.KafkaServerStartable)
> > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > at
> kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > at
> > > > >
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> 4)
> > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > java.net.BindException: Cannot assign requested address
> > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > at
> > > > >
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> 39)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > ... 6 more
> > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1],
> Shutting
> > > > > down
> > > > > (kafka.network.SocketServer)
> > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1],
> Shutdown
> > > > > completed (kafka.network.SocketServer)
> > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > (kafka.utils.KafkaScheduler)
> > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> completed
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > > (kafka.server.KafkaServer)
> > > > >
> > > > > Regards
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > You should should use the public IP for host.name. What's the
> error
> > > > > you see during broker startup?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I checked the faq. I did change the host.name in server
> properties.
> > > > > > After changing it I get ConnectException.
> > > > > >
> > > > > > The problem here is in EC2 have a different public IP
> > > > > > address
> > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > I set the host.name property to the local IP address which
> > > > > > is 10.x.x.x. I think because of this there is a ConnectException.
> > > > > > When I set the host.name to the public ip address
> > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > >
> > > > > > What should be the IP address that is to be given in
> > > host.nameproperty.
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Have you looked at
> > > > > >
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > ?
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > > > There are
> > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> zookeeper
> > > > > > > is running in the port 2181.
> > > > > > > When I send message to the brokers from my PC, I get an
> > > > > > > exception which is given below. I did a dump in the remote
> > > > > > > server, the request is received in the remote server.
> > > > > > > I am able to locally test the consumer/producer script
> > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > kindly help me in this
> > > > > error?
> > > > > > > Any help will be highly grateful.
> > > > > > >
> > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > metadata.broker.list is overridden to
> > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> overridden
> > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > 16:06:50,154]
> > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0
> > > > > > > for 1
> > > > > > > topic(s)
> > > > > > > Set(mytopic)
> > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > > SO_TIMEOUT =
> > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > > SO_SNDBUF =
> > > > > > > 102400 (requested 102400).
> > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > > 15:59:44,018] Disconnecting from
> > > > > > > 54.241.44.129:9093
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > fetched metadata for 1
> > > > > > > topic(s) Set(mytopic)
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > partition info for topic mytopic [DEBUG] [main 2014-01-27
> > > > > > > 15:59:44,060] Partition [mytopic,0] has leader 2 [DEBUG]
> > > > > > > [main 2014-01-27 15:59:44,072] Broker partitions
> > > > > > > registered for
> > > > > > > topic: mytopic are 0
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages
> > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > 2014-01-27 15:59:44,109] Producer sending messages with
> > > > > > > correlation id 2 for topics [mytopic,0] to broker 2 on
> > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection
> > > > > > > to
> > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > at
> > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > 640)
> > > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > )
> > > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > at
> > > > > > >
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > > )
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > nc
> > > > > > Pr
> > > > > > oducer.scala:68)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > scala:101)
> > > > > > > at
> > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > :1
> > > > > 01)
> > > > > > > at
> > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > :1
> > > > > 01)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > ul
> > > > > > tE
> > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > ed
> > > > > > Da
> > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > ed
> > > > > > Da
> > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > sc
> > > > > > al
> > > > > > a:80)
> > > > > > > at
> > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > sc
> > > > > > al
> > > > > > a:80)
> > > > > > > at
> > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > at
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > > )
> > > > > > > at
> > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > 194)
> > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > )
> > > > > > > at
> > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > fa
> > > > > > ul
> > > > > > tEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > sc
> > > > > > ala:72)
> > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > at
> > > > > > >
> kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > at
> > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:2
> > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to send
> > > > > > > producer request with correlation id 2 to broker 2 with
> > > > > > > data for partitions [mytopic,0]
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
After correcting the spelling, it is registered to correct public IP.
Thanks
Bala
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Thursday, 6 February, 2014 2:04 PM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
I think you mis-spelled the property name. It should be "
advertised.host.name", instead of advertise.host.name
Thanks,
Jun
On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> Thanks Joel.
>
> It seems the Broker is registered to the zookeeper with the IP
> 10.199.31.87. The output pf the command is given below.
>
> [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> /brokers/ids/1
> Connecting to 0.0.0.0:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
>
> {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","vers
> ion":1,"port":9092}
> cZxid = 0x2d
> ctime = Thu Feb 06 01:56:04 UTC 2014
> mZxid = 0x2d
> mtime = Thu Feb 06 01:56:04 UTC 2014
> pZxid = 0x2d
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x14404e371520000
> dataLength = 89
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> Sent: Thursday, February 06, 2014 5:56 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Use zookeeper-shell script:
>
> ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> /brokers/ids/<your broker id>
>
> On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman
> (Contingent) wrote:
> > Where should I look for these information. From the logs, I could
> > see
> ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is
> started in port 9082 and bind to IP 10.x.x.x.
> > If I don't give the host.name in server.properties, I get "
> java.nio.channels.UnresolvedAddressException" and If I give the host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > It is the same behavior as in 0.8.0.
> >
> > Thanks
> > Bala
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, February 05, 2014 12:21 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > It seems what's registered in ZK (10.199.31.87 <
> http://10.199.31.87:9094/>)
> > is still the local ip, not the public one. Could you check the
> > broker registration in zk (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structure
> s+in+Zookeeper
> )
> > and see what's the host/port of the broker?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I downloaded from the trunk and set up the properties host.name=
> > > <local IP> advertise.host.name=<public IP>
> > >
> > > Even after this change, I get the ConnectException. The details
> > > logs
> are
> > > given below. Is there any workaround for this?
> > >
> > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > 54.241.44.129:9094
> > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched
> > > metadata
> for 1
> > > topic(s) Set(mytopic)
> > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition
> > > info
> for
> > > topic mytopic
> > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has
> leader 3
> > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions
> > > registered for
> > > topic: mytopic are 0
> > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > > compression to [mytopic,0] [DEBUG] [main 2014-02-04 17:46:01,833]
> > > Producer sending messages with correlation id 2 for topics
> > > [mytopic,0] to broker 3 on
> 10.199.31.87:9094
> > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > 10.199.31.87:9094 unsuccessful
> > > java.net.ConnectException: Connection timed out: connect
> > > at sun.nio.ch.Net.connect0(Native Method)
> > > at sun.nio.ch.Net.connect(Net.java:465)
> > > at sun.nio.ch.Net.connect(Net.java:457)
> > > at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > at
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > at
> > >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncPr
> oducer.scala:68)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> ply$mcV$sp(SyncProducer.scala:102)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> ply(SyncProducer.scala:102)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.ap
> ply(SyncProducer.scala:102)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> scala:101)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> 01)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:1
> 01)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultE
> ventHandler$$send(DefaultEventHandler.scala:254)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> ta$1.apply(DefaultEventHandler.scala:106)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedDa
> ta$1.apply(DefaultEventHandler.scala:100)
> > > at
> > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> a:80)
> > > at
> > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scal
> a:80)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > at
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > > at
> > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> 194)
> > > at
> scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(Defaul
> tEventHandler.scala:100)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.sc
> ala:72)
> > > at kafka.producer.Producer.send(Producer.scala:76)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at
> > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.
> java:48)
> > > at
> > >
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(Kaf
> kaProducerTest.java:33)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:57)
> > > at
> > >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > > at junit.framework.TestResult.run(TestResult.java:109)
> > > at junit.framework.TestCase.run(TestCase.java:118)
> > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(J
> Unit3TestReference.java:131)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.
> java:38)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> TestRunner.java:467)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(Remote
> TestRunner.java:683)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestR
> unner.java:390)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTest
> Runner.java:197)
> > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer
> > > request with correlation id 2 to broker 3 with data for partitions
> > > [mytopic,0]
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Hmm, it's weird that EC2 only allows you to bind to local ip.
> > > Could
> some
> > > EC2 users here help out?
> > >
> > > Also, we recently added
> https://issues.apache.org/jira/browse/KAFKA-1092,
> > > which allows one to use a different ip for binding and connecting.
> > > You
> can
> > > see if this works for you. The patch is only in trunk though.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> (Contingent) <
> > > balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I don't think so. I forgot to include the ifconfig output.
> > > > Actually the public IP is not one of the IP configured in the
> > > > Ethernet
> interfaces.
> > > > Only the Local IP is configured in eth0.
> > > > Is there any solution to this?
> > > >
> > > > Ifconfig O/P:
> > > >
> > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > inet addr:10.X.X.X Bcast:10.199.31.127
> Mask:255.255.255.192
> > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > > > collisions:0 txqueuelen:1000
> > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> > > > Interrupt:25
> > > >
> > > > lo Link encap:Local Loopback
> > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > inet6 addr: ::1/128 Scope:Host
> > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > > > TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> > > > collisions:0 txqueuelen:0
> > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672
> > > > (104.0
> > > > MiB)
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Could it be a port conflict?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thanks for your help.
> > > > > I get the following exception :
> > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > at
> kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > at
> > > > >
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> 4)
> > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > java.net.BindException: Cannot assign requested address
> > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > at
> > > > >
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> 39)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > ... 6 more
> > > > >
> > > > > The entire stack trace of the logs are placed below.
> > > > >
> > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,176] INFO Property host.name is
> > > > > overridden to
> > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to
> > > > > 9092
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property
> > > > > socket.request.max.bytes is overridden to 104857600
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> overridden
> > > > > to 2
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden
> > > > > to
> > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > log.cleanup.interval.mins
> is
> > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property
> > > > > socket.send.buffer.bytes is overridden to 1048576
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms
> > > > > is overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > > overridden to
> > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is
> > > > > overridden
> to 1
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property
> > > > > socket.receive.buffer.bytes is overridden to 1048576
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> overridden
> > > > > to 2
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > log.flush.interval.messages is overridden to 10000
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > > overridden to
> > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1]
> > > > > Starting log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1]
> > > > > Starting log flusher every 3000 ms with the following
> > > > > overrides Map()
> > > > > (kafka.log.LogManager)
> > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> KafkaServerStable
> > > > > startup. Prepare to shutdown
> > > > > (kafka.server.KafkaServerStartable)
> > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > at
> kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > at
> > > > >
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:3
> 4)
> > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > java.net.BindException: Cannot assign requested address
> > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > at
> > > > >
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:1
> 39)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > ... 6 more
> > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1],
> Shutting
> > > > > down
> > > > > (kafka.network.SocketServer)
> > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1],
> Shutdown
> > > > > completed (kafka.network.SocketServer)
> > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > (kafka.utils.KafkaScheduler)
> > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> completed
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > > (kafka.server.KafkaServer)
> > > > >
> > > > > Regards
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > You should should use the public IP for host.name. What's the
> error
> > > > > you see during broker startup?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I checked the faq. I did change the host.name in server
> properties.
> > > > > > After changing it I get ConnectException.
> > > > > >
> > > > > > The problem here is in EC2 have a different public IP
> > > > > > address
> > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > I set the host.name property to the local IP address which
> > > > > > is 10.x.x.x. I think because of this there is a ConnectException.
> > > > > > When I set the host.name to the public ip address
> > > > > > (55.x.x.x), I cannot even start the broker.
> > > > > >
> > > > > > What should be the IP address that is to be given in
> > > host.nameproperty.
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Have you looked at
> > > > > >
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > ?
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > > > There are
> > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> zookeeper
> > > > > > > is running in the port 2181.
> > > > > > > When I send message to the brokers from my PC, I get an
> > > > > > > exception which is given below. I did a dump in the remote
> > > > > > > server, the request is received in the remote server.
> > > > > > > I am able to locally test the consumer/producer script
> > > > > > > present in the bin folder. What am I missing? Can you
> > > > > > > kindly help me in this
> > > > > error?
> > > > > > > Any help will be highly grateful.
> > > > > > >
> > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying
> > > > > > > properties [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > metadata.broker.list is overridden to
> > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> overridden
> > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > 16:06:50,154]
> > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0
> > > > > > > for 1
> > > > > > > topic(s)
> > > > > > > Set(mytopic)
> > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > > SO_TIMEOUT =
> > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > > SO_SNDBUF =
> > > > > > > 102400 (requested 102400).
> > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > > 15:59:44,018] Disconnecting from
> > > > > > > 54.241.44.129:9093
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully
> > > > > > > fetched metadata for 1
> > > > > > > topic(s) Set(mytopic)
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker
> > > > > > > partition info for topic mytopic [DEBUG] [main 2014-01-27
> > > > > > > 15:59:44,060] Partition [mytopic,0] has leader 2 [DEBUG]
> > > > > > > [main 2014-01-27 15:59:44,072] Broker partitions
> > > > > > > registered for
> > > > > > > topic: mytopic are 0
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages
> > > > > > > with no compression to [mytopic,0] [DEBUG] [main
> > > > > > > 2014-01-27 15:59:44,109] Producer sending messages with
> > > > > > > correlation id 2 for topics [mytopic,0] to broker 2 on
> > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection
> > > > > > > to
> > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > unsuccessful java.nio.channels.UnresolvedAddressException
> > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > at
> > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:
> > > > > > 640)
> > > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57
> > > > > )
> > > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > at
> > > > > > >
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > > )
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > nc
> > > > > > Pr
> > > > > > oducer.scala:68)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > scala:101)
> > > > > > > at
> > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > :1
> > > > > 01)
> > > > > > > at
> > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > :1
> > > > > 01)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > ul
> > > > > > tE
> > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > ed
> > > > > > Da
> > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > ed
> > > > > > Da
> > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > sc
> > > > > > al
> > > > > > a:80)
> > > > > > > at
> > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > sc
> > > > > > al
> > > > > > a:80)
> > > > > > > at
> > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > at
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > > )
> > > > > > > at
> > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > 194)
> > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39
> > > > > )
> > > > > > > at
> > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > fa
> > > > > > ul
> > > > > > tEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > sc
> > > > > > ala:72)
> > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > at
> > > > > > >
> kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > at
> > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:2
> > > > > > > 1) [ WARN] [main 2014-01-27 15:59:44,139] Failed to send
> > > > > > > producer request with correlation id 2 to broker 2 with
> > > > > > > data for partitions [mytopic,0]
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
Re: Reg Exception in Kafka
Posted by Jun Rao <ju...@gmail.com>.
I think you mis-spelled the property name. It should be "
advertised.host.name", instead of advertise.host.name
Thanks,
Jun
On Wed, Feb 5, 2014 at 6:06 PM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> Thanks Joel.
>
> It seems the Broker is registered to the zookeeper with the IP
> 10.199.31.87. The output pf the command is given below.
>
> [root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get
> /brokers/ids/1
> Connecting to 0.0.0.0:2181
>
> WATCHER::
>
> WatchedEvent state:SyncConnected type:None path:null
>
> {"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","version":1,"port":9092}
> cZxid = 0x2d
> ctime = Thu Feb 06 01:56:04 UTC 2014
> mZxid = 0x2d
> mtime = Thu Feb 06 01:56:04 UTC 2014
> pZxid = 0x2d
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x14404e371520000
> dataLength = 89
>
> Thanks
> Bala
>
> -----Original Message-----
> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> Sent: Thursday, February 06, 2014 5:56 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Use zookeeper-shell script:
>
> ./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get
> /brokers/ids/<your broker id>
>
> On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman
> (Contingent) wrote:
> > Where should I look for these information. From the logs, I could see
> ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is started
> in port 9082 and bind to IP 10.x.x.x.
> > If I don't give the host.name in server.properties, I get "
> java.nio.channels.UnresolvedAddressException" and If I give the host.nameto the local IP "10.x.x.x" I get "ConnectException.
> > It is the same behavior as in 0.8.0.
> >
> > Thanks
> > Bala
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, February 05, 2014 12:21 AM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > It seems what's registered in ZK (10.199.31.87 <
> http://10.199.31.87:9094/>)
> > is still the local ip, not the public one. Could you check the broker
> > registration in zk (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper
> )
> > and see what's the host/port of the broker?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman (Contingent) <
> > balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I downloaded from the trunk and set up the properties
> > > host.name= <local IP>
> > > advertise.host.name=<public IP>
> > >
> > > Even after this change, I get the ConnectException. The details logs
> are
> > > given below. Is there any workaround for this?
> > >
> > > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > > 54.241.44.129:9094
> > > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched metadata
> for 1
> > > topic(s) Set(mytopic)
> > > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition info
> for
> > > topic mytopic
> > > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has
> leader 3
> > > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions registered for
> > > topic: mytopic are 0
> > > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > > compression to [mytopic,0]
> > > [DEBUG] [main 2014-02-04 17:46:01,833] Producer sending messages with
> > > correlation id 2 for topics [mytopic,0] to broker 3 on
> 10.199.31.87:9094
> > > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > > 10.199.31.87:9094 unsuccessful
> > > java.net.ConnectException: Connection timed out: connect
> > > at sun.nio.ch.Net.connect0(Native Method)
> > > at sun.nio.ch.Net.connect(Net.java:465)
> > > at sun.nio.ch.Net.connect(Net.java:457)
> > > at
> sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > > at
> kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > at
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > > at
> > >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> > > at
> > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:254)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
> > > at
> > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> > > at
> > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> > > at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > at
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > > at
> > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
> > > at
> scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
> > > at
> > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> > > at kafka.producer.Producer.send(Producer.scala:76)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at
> > >
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> > > at
> > >
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > at
> > >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at junit.framework.TestCase.runTest(TestCase.java:154)
> > > at junit.framework.TestCase.runBare(TestCase.java:127)
> > > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > > at junit.framework.TestResult.run(TestResult.java:109)
> > > at junit.framework.TestCase.run(TestCase.java:118)
> > > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > > at junit.framework.TestSuite.run(TestSuite.java:203)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> > > at
> > >
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> > > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer request
> > > with correlation id 2 to broker 3 with data for partitions [mytopic,0]
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, January 29, 2014 11:12 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Hmm, it's weird that EC2 only allows you to bind to local ip. Could
> some
> > > EC2 users here help out?
> > >
> > > Also, we recently added
> https://issues.apache.org/jira/browse/KAFKA-1092,
> > > which allows one to use a different ip for binding and connecting. You
> can
> > > see if this works for you. The patch is only in trunk though.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman
> (Contingent) <
> > > balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I don't think so. I forgot to include the ifconfig output. Actually
> > > > the public IP is not one of the IP configured in the Ethernet
> interfaces.
> > > > Only the Local IP is configured in eth0.
> > > > Is there any solution to this?
> > > >
> > > > Ifconfig O/P:
> > > >
> > > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > > inet addr:10.X.X.X Bcast:10.199.31.127
> Mask:255.255.255.192
> > > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > > > collisions:0 txqueuelen:1000
> > > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> > > > Interrupt:25
> > > >
> > > > lo Link encap:Local Loopback
> > > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > > inet6 addr: ::1/128 Scope:Host
> > > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > > > TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> > > > collisions:0 txqueuelen:0
> > > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672 (104.0
> > > > MiB)
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Could it be a port conflict?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > Jun,
> > > > >
> > > > > Thanks for your help.
> > > > > I get the following exception :
> > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > at
> kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > at
> > > > >
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > java.net.BindException: Cannot assign requested address
> > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > at
> > > > >
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > ... 6 more
> > > > >
> > > > > The entire stack trace of the logs are placed below.
> > > > >
> > > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,176] INFO Property host.name is overridden to
> > > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to 9092
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property socket.request.max.bytes is
> > > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is
> overridden
> > > > > to 2
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden to
> > > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property log.cleanup.interval.mins
> is
> > > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,178] INFO Property socket.send.buffer.bytes is
> > > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms is
> > > > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > > overridden to
> > > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property broker.id is overridden
> to 1
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,180] INFO Property socket.receive.buffer.bytes
> > > > > is overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property
> > > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is
> overridden
> > > > > to 2
> > > > > (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,181] INFO Property log.flush.interval.messages
> > > > > is overridden to 10000 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > > overridden to
> > > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1] Starting
> > > > > log cleaner every 60000 ms (kafka.log.LogManager)
> > > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1] Starting
> > > > > log flusher every 3000 ms with the following overrides Map()
> > > > > (kafka.log.LogManager)
> > > > > [2014-01-29 01:18:23,330] FATAL Fatal error during
> KafkaServerStable
> > > > > startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> > > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > > at
> kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > > at
> > > > >
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > > java.net.BindException: Cannot assign requested address
> > > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > > at
> > > > >
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > > at
> > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > > at
> > > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > > ... 6 more
> > > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1],
> Shutting
> > > > > down
> > > > > (kafka.network.SocketServer)
> > > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1],
> Shutdown
> > > > > completed (kafka.network.SocketServer)
> > > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > > (kafka.utils.KafkaScheduler)
> > > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down
> completed
> > > > > (kafka.server.KafkaServer)
> > > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > > (kafka.server.KafkaServer)
> > > > >
> > > > > Regards
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > You should should use the public IP for host.name. What's the
> error
> > > > > you see during broker startup?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > I checked the faq. I did change the host.name in server
> properties.
> > > > > > After changing it I get ConnectException.
> > > > > >
> > > > > > The problem here is in EC2 have a different public IP address
> > > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > > I set the host.name property to the local IP address which is
> > > > > > 10.x.x.x. I think because of this there is a ConnectException.
> > > > > > When I set the host.name to the public ip address (55.x.x.x), I
> > > > > > cannot even start the broker.
> > > > > >
> > > > > > What should be the IP address that is to be given in
> > > host.nameproperty.
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > > To: users@kafka.apache.org
> > > > > > Subject: Re: Reg Exception in Kafka
> > > > > >
> > > > > > Have you looked at
> > > > > >
> > > > > >
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > > ?
> > > > > >
> > > > > > thanks,
> > > > > >
> > > > > > Jun
> > > > > >
> > > > > >
> > > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > > > There are
> > > > > > > 3 brokers each running in the port 9092,9093,9094. The
> zookeeper
> > > > > > > is running in the port 2181.
> > > > > > > When I send message to the brokers from my PC, I get an
> > > > > > > exception which is given below. I did a dump in the remote
> > > > > > > server, the request is received in the remote server.
> > > > > > > I am able to locally test the consumer/producer script present
> > > > > > > in the bin folder. What am I missing? Can you kindly help me in
> > > > > > > this
> > > > > error?
> > > > > > > Any help will be highly grateful.
> > > > > > >
> > > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying properties [
> > > > > > > INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > metadata.broker.list is overridden to
> > > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > > 2014-01-27 16:06:50,108] Property serializer.class is
> overridden
> > > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > > 16:06:50,154]
> > > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0 for 1
> > > > > > > topic(s)
> > > > > > > Set(mytopic)
> > > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > > SO_TIMEOUT =
> > > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > > SO_SNDBUF =
> > > > > > > 102400 (requested 102400).
> > > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > > 15:59:44,018] Disconnecting from
> > > > > > > 54.241.44.129:9093
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully fetched
> > > > > > > metadata for 1
> > > > > > > topic(s) Set(mytopic)
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker partition
> > > > > > > info for topic mytopic [DEBUG] [main 2014-01-27 15:59:44,060]
> > > > > > > Partition [mytopic,0] has leader 2 [DEBUG] [main 2014-01-27
> > > > > > > 15:59:44,072] Broker partitions registered for
> > > > > > > topic: mytopic are 0
> > > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages with
> > > > > > > no compression to [mytopic,0] [DEBUG] [main 2014-01-27
> > > > > > > 15:59:44,109] Producer sending messages with correlation id 2
> > > > > > > for topics [mytopic,0] to broker 2 on
> > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection to
> > > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093 unsuccessful
> > > > > > > java.nio.channels.UnresolvedAddressException
> > > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > > at
> > > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
> > > > > > > at
> > > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > > at
> > > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > > at
> > > > > > >
> > > >
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > > )
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > > nc
> > > > > > Pr
> > > > > > oducer.scala:68)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > > ap
> > > > > > ply(SyncProducer.scala:102)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> > > > > > >
> > > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > > scala:101)
> > > > > > > at
> > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > :1
> > > > > 01)
> > > > > > > at
> > > > > > >
> > > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > > :1
> > > > > 01)
> > > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > > at
> kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > > ul
> > > > > > tE
> > > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > ed
> > > > > > Da
> > > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > > ed
> > > > > > Da
> > > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > sc
> > > > > > al
> > > > > > a:80)
> > > > > > > at
> > > > > > >
> > > > > >
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > > sc
> > > > > > al
> > > > > > a:80)
> > > > > > > at
> > > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > > at
> > > > > > >
> > > >
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > > )
> > > > > > > at
> > > > > > >
> > > > > >
> > > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > > 194)
> > > > > > > at
> > > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > > at
> > > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > > fa
> > > > > > ul
> > > > > > tEventHandler.scala:100)
> > > > > > > at
> > > > > > >
> > > > > >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > > sc
> > > > > > ala:72)
> > > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > > at
> kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > > at
> > > > > > >
> kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > > at
> > > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:21)
> > > > > > > [ WARN] [main 2014-01-27 15:59:44,139] Failed to send producer
> > > > > > > request with correlation id 2 to broker 2 with data for
> > > > > > > partitions [mytopic,0]
> > > > > > >
> > > > > > > Thanks
> > > > > > > Bala
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
>
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
Thanks Joel.
It seems the Broker is registered to the zookeeper with the IP 10.199.31.87. The output pf the command is given below.
[root@ip-10-199-31-87 bin]# ./zookeeper-shell.sh 0.0.0.0:2181 get /brokers/ids/1
Connecting to 0.0.0.0:2181
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
{"jmx_port":-1,"timestamp":"1391651764357","host":"10.199.31.87","version":1,"port":9092}
cZxid = 0x2d
ctime = Thu Feb 06 01:56:04 UTC 2014
mZxid = 0x2d
mtime = Thu Feb 06 01:56:04 UTC 2014
pZxid = 0x2d
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x14404e371520000
dataLength = 89
Thanks
Bala
-----Original Message-----
From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
Sent: Thursday, February 06, 2014 5:56 AM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
Use zookeeper-shell script:
./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get /brokers/ids/<your broker id>
On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman (Contingent) wrote:
> Where should I look for these information. From the logs, I could see ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is started in port 9082 and bind to IP 10.x.x.x.
> If I don't give the host.name in server.properties, I get " java.nio.channels.UnresolvedAddressException" and If I give the host.name to the local IP "10.x.x.x" I get "ConnectException.
> It is the same behavior as in 0.8.0.
>
> Thanks
> Bala
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, February 05, 2014 12:21 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> It seems what's registered in ZK (10.199.31.87 <http://10.199.31.87:9094/>)
> is still the local ip, not the public one. Could you check the broker
> registration in zk (
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper)
> and see what's the host/port of the broker?
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I downloaded from the trunk and set up the properties
> > host.name= <local IP>
> > advertise.host.name=<public IP>
> >
> > Even after this change, I get the ConnectException. The details logs are
> > given below. Is there any workaround for this?
> >
> > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > 54.241.44.129:9094
> > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched metadata for 1
> > topic(s) Set(mytopic)
> > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition info for
> > topic mytopic
> > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has leader 3
> > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions registered for
> > topic: mytopic are 0
> > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > compression to [mytopic,0]
> > [DEBUG] [main 2014-02-04 17:46:01,833] Producer sending messages with
> > correlation id 2 for topics [mytopic,0] to broker 3 on 10.199.31.87:9094
> > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > 10.199.31.87:9094 unsuccessful
> > java.net.ConnectException: Connection timed out: connect
> > at sun.nio.ch.Net.connect0(Native Method)
> > at sun.nio.ch.Net.connect(Net.java:465)
> > at sun.nio.ch.Net.connect(Net.java:457)
> > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > at
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > at
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > at
> > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:254)
> > at
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
> > at
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > at
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > at
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > at
> > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
> > at
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> > at kafka.producer.Producer.send(Producer.scala:76)
> > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > at
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> > at
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at junit.framework.TestCase.runTest(TestCase.java:154)
> > at junit.framework.TestCase.runBare(TestCase.java:127)
> > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > at junit.framework.TestResult.run(TestResult.java:109)
> > at junit.framework.TestCase.run(TestCase.java:118)
> > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > at junit.framework.TestSuite.run(TestSuite.java:203)
> > at
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> > at
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer request
> > with correlation id 2 to broker 3 with data for partitions [mytopic,0]
> >
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, January 29, 2014 11:12 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Hmm, it's weird that EC2 only allows you to bind to local ip. Could some
> > EC2 users here help out?
> >
> > Also, we recently added https://issues.apache.org/jira/browse/KAFKA-1092,
> > which allows one to use a different ip for binding and connecting. You can
> > see if this works for you. The patch is only in trunk though.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman (Contingent) <
> > balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I don't think so. I forgot to include the ifconfig output. Actually
> > > the public IP is not one of the IP configured in the Ethernet interfaces.
> > > Only the Local IP is configured in eth0.
> > > Is there any solution to this?
> > >
> > > Ifconfig O/P:
> > >
> > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > inet addr:10.X.X.X Bcast:10.199.31.127 Mask:255.255.255.192
> > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > > collisions:0 txqueuelen:1000
> > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> > > Interrupt:25
> > >
> > > lo Link encap:Local Loopback
> > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > inet6 addr: ::1/128 Scope:Host
> > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > > TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> > > collisions:0 txqueuelen:0
> > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672 (104.0
> > > MiB)
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Could it be a port conflict?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > Thanks for your help.
> > > > I get the following exception :
> > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > at
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > java.net.BindException: Cannot assign requested address
> > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > at
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > ... 6 more
> > > >
> > > > The entire stack trace of the logs are placed below.
> > > >
> > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,176] INFO Property host.name is overridden to
> > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to 9092
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,177] INFO Property socket.request.max.bytes is
> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is overridden
> > > > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden to
> > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,178] INFO Property log.cleanup.interval.mins is
> > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,178] INFO Property socket.send.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms is
> > > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > overridden to
> > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property broker.id is overridden to 1
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property socket.receive.buffer.bytes
> > > > is overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,181] INFO Property
> > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is overridden
> > > > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,181] INFO Property log.flush.interval.messages
> > > > is overridden to 10000 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > overridden to
> > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > (kafka.server.KafkaServer)
> > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1] Starting
> > > > log cleaner every 60000 ms (kafka.log.LogManager)
> > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1] Starting
> > > > log flusher every 3000 ms with the following overrides Map()
> > > > (kafka.log.LogManager)
> > > > [2014-01-29 01:18:23,330] FATAL Fatal error during KafkaServerStable
> > > > startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > at
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > java.net.BindException: Cannot assign requested address
> > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > at
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > ... 6 more
> > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > (kafka.server.KafkaServer)
> > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1], Shutting
> > > > down
> > > > (kafka.network.SocketServer)
> > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1], Shutdown
> > > > completed (kafka.network.SocketServer)
> > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > (kafka.utils.KafkaScheduler)
> > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down completed
> > > > (kafka.server.KafkaServer)
> > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > (kafka.server.KafkaServer)
> > > >
> > > > Regards
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > You should should use the public IP for host.name. What's the error
> > > > you see during broker startup?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I checked the faq. I did change the host.name in server properties.
> > > > > After changing it I get ConnectException.
> > > > >
> > > > > The problem here is in EC2 have a different public IP address
> > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > I set the host.name property to the local IP address which is
> > > > > 10.x.x.x. I think because of this there is a ConnectException.
> > > > > When I set the host.name to the public ip address (55.x.x.x), I
> > > > > cannot even start the broker.
> > > > >
> > > > > What should be the IP address that is to be given in
> > host.nameproperty.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Have you looked at
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > ?
> > > > >
> > > > > thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > > There are
> > > > > > 3 brokers each running in the port 9092,9093,9094. The zookeeper
> > > > > > is running in the port 2181.
> > > > > > When I send message to the brokers from my PC, I get an
> > > > > > exception which is given below. I did a dump in the remote
> > > > > > server, the request is received in the remote server.
> > > > > > I am able to locally test the consumer/producer script present
> > > > > > in the bin folder. What am I missing? Can you kindly help me in
> > > > > > this
> > > > error?
> > > > > > Any help will be highly grateful.
> > > > > >
> > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying properties [
> > > > > > INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > metadata.broker.list is overridden to
> > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > 2014-01-27 16:06:50,108] Property serializer.class is overridden
> > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > 16:06:50,154]
> > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0 for 1
> > > > > > topic(s)
> > > > > > Set(mytopic)
> > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > SO_TIMEOUT =
> > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > SO_SNDBUF =
> > > > > > 102400 (requested 102400).
> > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > 15:59:44,018] Disconnecting from
> > > > > > 54.241.44.129:9093
> > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully fetched
> > > > > > metadata for 1
> > > > > > topic(s) Set(mytopic)
> > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker partition
> > > > > > info for topic mytopic [DEBUG] [main 2014-01-27 15:59:44,060]
> > > > > > Partition [mytopic,0] has leader 2 [DEBUG] [main 2014-01-27
> > > > > > 15:59:44,072] Broker partitions registered for
> > > > > > topic: mytopic are 0
> > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages with
> > > > > > no compression to [mytopic,0] [DEBUG] [main 2014-01-27
> > > > > > 15:59:44,109] Producer sending messages with correlation id 2
> > > > > > for topics [mytopic,0] to broker 2 on
> > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection to
> > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093 unsuccessful
> > > > > > java.nio.channels.UnresolvedAddressException
> > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > at
> > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
> > > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > at
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > )
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > nc
> > > > > Pr
> > > > > oducer.scala:68)
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > ap
> > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> > > > > >
> > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > scala:101)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > :1
> > > > 01)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > :1
> > > > 01)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > ul
> > > > > tE
> > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > ed
> > > > > Da
> > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > ed
> > > > > Da
> > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > sc
> > > > > al
> > > > > a:80)
> > > > > > at
> > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > sc
> > > > > al
> > > > > a:80)
> > > > > > at
> > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > at
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > )
> > > > > > at
> > > > > >
> > > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > 194)
> > > > > > at
> > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > at
> > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > fa
> > > > > ul
> > > > > tEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > sc
> > > > > ala:72)
> > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > at
> > > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > at
> > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:21)
> > > > > > [ WARN] [main 2014-01-27 15:59:44,139] Failed to send producer
> > > > > > request with correlation id 2 to broker 2 with data for
> > > > > > partitions [mytopic,0]
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > >
> > > >
> > >
> >
Re: Reg Exception in Kafka
Posted by Joel Koshy <jj...@gmail.com>.
Use zookeeper-shell script:
./bin/zookeeper-shell.sh <zkhost>:<zkport></namespace> get /brokers/ids/<your broker id>
On Wed, Feb 05, 2014 at 07:04:50AM +0000, Balasubramanian Jayaraman (Contingent) wrote:
> Where should I look for these information. From the logs, I could see ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is started in port 9082 and bind to IP 10.x.x.x.
> If I don't give the host.name in server.properties, I get " java.nio.channels.UnresolvedAddressException" and If I give the host.name to the local IP "10.x.x.x" I get "ConnectException.
> It is the same behavior as in 0.8.0.
>
> Thanks
> Bala
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, February 05, 2014 12:21 AM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> It seems what's registered in ZK (10.199.31.87 <http://10.199.31.87:9094/>)
> is still the local ip, not the public one. Could you check the broker
> registration in zk (
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper)
> and see what's the host/port of the broker?
>
> Thanks,
>
> Jun
>
>
> On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I downloaded from the trunk and set up the properties
> > host.name= <local IP>
> > advertise.host.name=<public IP>
> >
> > Even after this change, I get the ConnectException. The details logs are
> > given below. Is there any workaround for this?
> >
> > [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> > 54.241.44.129:9094
> > [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched metadata for 1
> > topic(s) Set(mytopic)
> > [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition info for
> > topic mytopic
> > [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has leader 3
> > [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions registered for
> > topic: mytopic are 0
> > [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> > compression to [mytopic,0]
> > [DEBUG] [main 2014-02-04 17:46:01,833] Producer sending messages with
> > correlation id 2 for topics [mytopic,0] to broker 3 on 10.199.31.87:9094
> > [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> > 10.199.31.87:9094 unsuccessful
> > java.net.ConnectException: Connection timed out: connect
> > at sun.nio.ch.Net.connect0(Native Method)
> > at sun.nio.ch.Net.connect(Net.java:465)
> > at sun.nio.ch.Net.connect(Net.java:457)
> > at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> > at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > at
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> > at
> > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> > at
> > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > at
> > kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:254)
> > at
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
> > at
> > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> > at
> > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> > at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > at
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> > at
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
> > at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > at
> > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
> > at
> > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> > at kafka.producer.Producer.send(Producer.scala:76)
> > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > at
> > com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> > at
> > com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > at
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:606)
> > at junit.framework.TestCase.runTest(TestCase.java:154)
> > at junit.framework.TestCase.runBare(TestCase.java:127)
> > at junit.framework.TestResult$1.protect(TestResult.java:106)
> > at junit.framework.TestResult.runProtected(TestResult.java:124)
> > at junit.framework.TestResult.run(TestResult.java:109)
> > at junit.framework.TestCase.run(TestCase.java:118)
> > at junit.framework.TestSuite.runTest(TestSuite.java:208)
> > at junit.framework.TestSuite.run(TestSuite.java:203)
> > at
> > org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> > at
> > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> > at
> > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> > [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer request
> > with correlation id 2 to broker 3 with data for partitions [mytopic,0]
> >
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, January 29, 2014 11:12 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Hmm, it's weird that EC2 only allows you to bind to local ip. Could some
> > EC2 users here help out?
> >
> > Also, we recently added https://issues.apache.org/jira/browse/KAFKA-1092,
> > which allows one to use a different ip for binding and connecting. You can
> > see if this works for you. The patch is only in trunk though.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman (Contingent) <
> > balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > I don't think so. I forgot to include the ifconfig output. Actually
> > > the public IP is not one of the IP configured in the Ethernet interfaces.
> > > Only the Local IP is configured in eth0.
> > > Is there any solution to this?
> > >
> > > Ifconfig O/P:
> > >
> > > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > > inet addr:10.X.X.X Bcast:10.199.31.127 Mask:255.255.255.192
> > > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > > collisions:0 txqueuelen:1000
> > > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> > > Interrupt:25
> > >
> > > lo Link encap:Local Loopback
> > > inet addr:127.0.0.1 Mask:255.0.0.0
> > > inet6 addr: ::1/128 Scope:Host
> > > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > > TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> > > collisions:0 txqueuelen:0
> > > RX bytes:109133672 (104.0 MiB) TX bytes:109133672 (104.0
> > > MiB)
> > >
> > > Thanks
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Wednesday, January 29, 2014 12:27 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > Could it be a port conflict?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > Jun,
> > > >
> > > > Thanks for your help.
> > > > I get the following exception :
> > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > at
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > java.net.BindException: Cannot assign requested address
> > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > at
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > ... 6 more
> > > >
> > > > The entire stack trace of the logs are placed below.
> > > >
> > > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,176] INFO Property host.name is overridden to
> > > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,177] INFO Property port is overridden to 9092
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,177] INFO Property socket.request.max.bytes is
> > > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is overridden
> > > > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden to
> > > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,178] INFO Property log.cleanup.interval.mins is
> > > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,178] INFO Property socket.send.buffer.bytes is
> > > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms is
> > > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > > overridden to
> > > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property broker.id is overridden to 1
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,180] INFO Property socket.receive.buffer.bytes
> > > > is overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,181] INFO Property
> > > > zookeeper.connection.timeout.msis overridden to 1000000
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,181] INFO Property num.partitions is overridden
> > > > to 2
> > > > (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,181] INFO Property log.flush.interval.messages
> > > > is overridden to 10000 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > > overridden to
> > > > 536870912 (kafka.utils.VerifiableProperties)
> > > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > > (kafka.server.KafkaServer)
> > > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1] Starting
> > > > log cleaner every 60000 ms (kafka.log.LogManager)
> > > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1] Starting
> > > > log flusher every 3000 ms with the following overrides Map()
> > > > (kafka.log.LogManager)
> > > > [2014-01-29 01:18:23,330] FATAL Fatal error during KafkaServerStable
> > > > startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> > > > kafka.common.KafkaException: Socket server failed to bind to
> > > > 54.241.44.129:9092: Cannot assign requested address.
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > > at
> > > >
> > kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > > at kafka.Kafka$.main(Kafka.scala:46)
> > > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > > java.net.BindException: Cannot assign requested address
> > > > at sun.nio.ch.Net.bind0(Native Method)
> > > > at sun.nio.ch.Net.bind(Net.java:174)
> > > > at
> > > >
> > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > > at
> > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > > at
> > > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > > ... 6 more
> > > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > > (kafka.server.KafkaServer)
> > > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1], Shutting
> > > > down
> > > > (kafka.network.SocketServer)
> > > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1], Shutdown
> > > > completed (kafka.network.SocketServer)
> > > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > > (kafka.utils.KafkaScheduler)
> > > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down completed
> > > > (kafka.server.KafkaServer)
> > > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > > (kafka.server.KafkaServer)
> > > >
> > > > Regards
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > You should should use the public IP for host.name. What's the error
> > > > you see during broker startup?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > I checked the faq. I did change the host.name in server properties.
> > > > > After changing it I get ConnectException.
> > > > >
> > > > > The problem here is in EC2 have a different public IP address
> > > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > > I set the host.name property to the local IP address which is
> > > > > 10.x.x.x. I think because of this there is a ConnectException.
> > > > > When I set the host.name to the public ip address (55.x.x.x), I
> > > > > cannot even start the broker.
> > > > >
> > > > > What should be the IP address that is to be given in
> > host.nameproperty.
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > > To: users@kafka.apache.org
> > > > > Subject: Re: Reg Exception in Kafka
> > > > >
> > > > > Have you looked at
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > > ?
> > > > >
> > > > > thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > >
> > > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > > There are
> > > > > > 3 brokers each running in the port 9092,9093,9094. The zookeeper
> > > > > > is running in the port 2181.
> > > > > > When I send message to the brokers from my PC, I get an
> > > > > > exception which is given below. I did a dump in the remote
> > > > > > server, the request is received in the remote server.
> > > > > > I am able to locally test the consumer/producer script present
> > > > > > in the bin folder. What am I missing? Can you kindly help me in
> > > > > > this
> > > > error?
> > > > > > Any help will be highly grateful.
> > > > > >
> > > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying properties [
> > > > > > INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > metadata.broker.list is overridden to
> > > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > > 2014-01-27 16:06:50,108] Property serializer.class is overridden
> > > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > > 16:06:50,154]
> > > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0 for 1
> > > > > > topic(s)
> > > > > > Set(mytopic)
> > > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > > SO_TIMEOUT =
> > > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > > SO_SNDBUF =
> > > > > > 102400 (requested 102400).
> > > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > > 15:59:44,018] Disconnecting from
> > > > > > 54.241.44.129:9093
> > > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully fetched
> > > > > > metadata for 1
> > > > > > topic(s) Set(mytopic)
> > > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker partition
> > > > > > info for topic mytopic [DEBUG] [main 2014-01-27 15:59:44,060]
> > > > > > Partition [mytopic,0] has leader 2 [DEBUG] [main 2014-01-27
> > > > > > 15:59:44,072] Broker partitions registered for
> > > > > > topic: mytopic are 0
> > > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages with
> > > > > > no compression to [mytopic,0] [DEBUG] [main 2014-01-27
> > > > > > 15:59:44,109] Producer sending messages with correlation id 2
> > > > > > for topics [mytopic,0] to broker 2 on
> > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection to
> > > > > > ip-10-199-31-87.us-west-1.compute.internal:9093 unsuccessful
> > > > > > java.nio.channels.UnresolvedAddressException
> > > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > > at
> > > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
> > > > > > at
> > > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > > at
> > kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > > at
> > > > > >
> > > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > > )
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > > nc
> > > > > Pr
> > > > > oducer.scala:68)
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > ap
> > > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > at
> > > > > >
> > > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > > ap
> > > > > ply(SyncProducer.scala:102)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at
> > > > > >
> > > > >
> > kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > > scala:101)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > :1
> > > > 01)
> > > > > > at
> > > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > > :1
> > > > 01)
> > > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > > ul
> > > > > tE
> > > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > ed
> > > > > Da
> > > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > > ed
> > > > > Da
> > > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > sc
> > > > > al
> > > > > a:80)
> > > > > > at
> > > > > >
> > > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > > sc
> > > > > al
> > > > > a:80)
> > > > > > at
> > scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > > at
> > > > > >
> > > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > > )
> > > > > > at
> > > > > >
> > > > >
> > scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > > 194)
> > > > > > at
> > > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > > at
> > scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > > fa
> > > > > ul
> > > > > tEventHandler.scala:100)
> > > > > > at
> > > > > >
> > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > > sc
> > > > > ala:72)
> > > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > > at
> > > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > > at
> > > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:21)
> > > > > > [ WARN] [main 2014-01-27 15:59:44,139] Failed to send producer
> > > > > > request with correlation id 2 to broker 2 with data for
> > > > > > partitions [mytopic,0]
> > > > > >
> > > > > > Thanks
> > > > > > Bala
> > > > > >
> > > > >
> > > >
> > >
> >
RE: Reg Exception in Kafka
Posted by "Balasubramanian Jayaraman (Contingent)" <ba...@autodesk.com>.
Where should I look for these information. From the logs, I could see ZooKeeper is bound to port 2181 and IP 0.0.0.0. The Kafka Server is started in port 9082 and bind to IP 10.x.x.x.
If I don't give the host.name in server.properties, I get " java.nio.channels.UnresolvedAddressException" and If I give the host.name to the local IP "10.x.x.x" I get "ConnectException.
It is the same behavior as in 0.8.0.
Thanks
Bala
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Wednesday, February 05, 2014 12:21 AM
To: users@kafka.apache.org
Subject: Re: Reg Exception in Kafka
It seems what's registered in ZK (10.199.31.87 <http://10.199.31.87:9094/>)
is still the local ip, not the public one. Could you check the broker
registration in zk (
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper)
and see what's the host/port of the broker?
Thanks,
Jun
On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> I downloaded from the trunk and set up the properties
> host.name= <local IP>
> advertise.host.name=<public IP>
>
> Even after this change, I get the ConnectException. The details logs are
> given below. Is there any workaround for this?
>
> [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> 54.241.44.129:9094
> [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched metadata for 1
> topic(s) Set(mytopic)
> [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition info for
> topic mytopic
> [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has leader 3
> [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions registered for
> topic: mytopic are 0
> [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> compression to [mytopic,0]
> [DEBUG] [main 2014-02-04 17:46:01,833] Producer sending messages with
> correlation id 2 for topics [mytopic,0] to broker 3 on 10.199.31.87:9094
> [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> 10.199.31.87:9094 unsuccessful
> java.net.ConnectException: Connection timed out: connect
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> at
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
> at
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> at
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
> at
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> at
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> at
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:254)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> at
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> at
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> at kafka.producer.Producer.send(Producer.scala:76)
> at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> at
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> at
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:154)
> at junit.framework.TestCase.runBare(TestCase.java:127)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:118)
> at junit.framework.TestSuite.runTest(TestSuite.java:208)
> at junit.framework.TestSuite.run(TestSuite.java:203)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer request
> with correlation id 2 to broker 3 with data for partitions [mytopic,0]
>
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, January 29, 2014 11:12 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Hmm, it's weird that EC2 only allows you to bind to local ip. Could some
> EC2 users here help out?
>
> Also, we recently added https://issues.apache.org/jira/browse/KAFKA-1092,
> which allows one to use a different ip for binding and connecting. You can
> see if this works for you. The patch is only in trunk though.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I don't think so. I forgot to include the ifconfig output. Actually
> > the public IP is not one of the IP configured in the Ethernet interfaces.
> > Only the Local IP is configured in eth0.
> > Is there any solution to this?
> >
> > Ifconfig O/P:
> >
> > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > inet addr:10.X.X.X Bcast:10.199.31.127 Mask:255.255.255.192
> > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> > Interrupt:25
> >
> > lo Link encap:Local Loopback
> > inet addr:127.0.0.1 Mask:255.0.0.0
> > inet6 addr: ::1/128 Scope:Host
> > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:109133672 (104.0 MiB) TX bytes:109133672 (104.0
> > MiB)
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, January 29, 2014 12:27 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Could it be a port conflict?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > Jun,
> > >
> > > Thanks for your help.
> > > I get the following exception :
> > > kafka.common.KafkaException: Socket server failed to bind to
> > > 54.241.44.129:9092: Cannot assign requested address.
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > at
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > at kafka.Kafka$.main(Kafka.scala:46)
> > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > java.net.BindException: Cannot assign requested address
> > > at sun.nio.ch.Net.bind0(Native Method)
> > > at sun.nio.ch.Net.bind(Net.java:174)
> > > at
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > ... 6 more
> > >
> > > The entire stack trace of the logs are placed below.
> > >
> > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,176] INFO Property host.name is overridden to
> > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,177] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,177] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is overridden
> > > to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,178] INFO Property log.cleanup.interval.mins is
> > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,178] INFO Property socket.send.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms is
> > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > overridden to
> > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property broker.id is overridden to 1
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property socket.receive.buffer.bytes
> > > is overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,181] INFO Property
> > > zookeeper.connection.timeout.msis overridden to 1000000
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,181] INFO Property num.partitions is overridden
> > > to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,181] INFO Property log.flush.interval.messages
> > > is overridden to 10000 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > overridden to
> > > 536870912 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > (kafka.server.KafkaServer)
> > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1] Starting
> > > log cleaner every 60000 ms (kafka.log.LogManager)
> > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1] Starting
> > > log flusher every 3000 ms with the following overrides Map()
> > > (kafka.log.LogManager)
> > > [2014-01-29 01:18:23,330] FATAL Fatal error during KafkaServerStable
> > > startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> > > kafka.common.KafkaException: Socket server failed to bind to
> > > 54.241.44.129:9092: Cannot assign requested address.
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > at
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > at kafka.Kafka$.main(Kafka.scala:46)
> > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > java.net.BindException: Cannot assign requested address
> > > at sun.nio.ch.Net.bind0(Native Method)
> > > at sun.nio.ch.Net.bind(Net.java:174)
> > > at
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > ... 6 more
> > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > (kafka.server.KafkaServer)
> > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1], Shutting
> > > down
> > > (kafka.network.SocketServer)
> > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1], Shutdown
> > > completed (kafka.network.SocketServer)
> > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > (kafka.utils.KafkaScheduler)
> > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down completed
> > > (kafka.server.KafkaServer)
> > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > (kafka.server.KafkaServer)
> > >
> > > Regards
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > You should should use the public IP for host.name. What's the error
> > > you see during broker startup?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I checked the faq. I did change the host.name in server properties.
> > > > After changing it I get ConnectException.
> > > >
> > > > The problem here is in EC2 have a different public IP address
> > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > I set the host.name property to the local IP address which is
> > > > 10.x.x.x. I think because of this there is a ConnectException.
> > > > When I set the host.name to the public ip address (55.x.x.x), I
> > > > cannot even start the broker.
> > > >
> > > > What should be the IP address that is to be given in
> host.nameproperty.
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Have you looked at
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > ?
> > > >
> > > > thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > There are
> > > > > 3 brokers each running in the port 9092,9093,9094. The zookeeper
> > > > > is running in the port 2181.
> > > > > When I send message to the brokers from my PC, I get an
> > > > > exception which is given below. I did a dump in the remote
> > > > > server, the request is received in the remote server.
> > > > > I am able to locally test the consumer/producer script present
> > > > > in the bin folder. What am I missing? Can you kindly help me in
> > > > > this
> > > error?
> > > > > Any help will be highly grateful.
> > > > >
> > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying properties [
> > > > > INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > metadata.broker.list is overridden to
> > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > 2014-01-27 16:06:50,108] Property serializer.class is overridden
> > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > 16:06:50,154]
> > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0 for 1
> > > > > topic(s)
> > > > > Set(mytopic)
> > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > SO_TIMEOUT =
> > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > SO_SNDBUF =
> > > > > 102400 (requested 102400).
> > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > 15:59:44,018] Disconnecting from
> > > > > 54.241.44.129:9093
> > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully fetched
> > > > > metadata for 1
> > > > > topic(s) Set(mytopic)
> > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker partition
> > > > > info for topic mytopic [DEBUG] [main 2014-01-27 15:59:44,060]
> > > > > Partition [mytopic,0] has leader 2 [DEBUG] [main 2014-01-27
> > > > > 15:59:44,072] Broker partitions registered for
> > > > > topic: mytopic are 0
> > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages with
> > > > > no compression to [mytopic,0] [DEBUG] [main 2014-01-27
> > > > > 15:59:44,109] Producer sending messages with correlation id 2
> > > > > for topics [mytopic,0] to broker 2 on
> > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection to
> > > > > ip-10-199-31-87.us-west-1.compute.internal:9093 unsuccessful
> > > > > java.nio.channels.UnresolvedAddressException
> > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
> > > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > at
> kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > at
> > > > >
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > )
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > nc
> > > > Pr
> > > > oducer.scala:68)
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > ap
> > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > ap
> > > > ply(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > ap
> > > > ply(SyncProducer.scala:102)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at
> > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > scala:101)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > :1
> > > 01)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > :1
> > > 01)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > ul
> > > > tE
> > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > ed
> > > > Da
> > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > ed
> > > > Da
> > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > at
> > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > sc
> > > > al
> > > > a:80)
> > > > > at
> > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > sc
> > > > al
> > > > a:80)
> > > > > at
> scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > at
> > > > >
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > )
> > > > > at
> > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > 194)
> > > > > at
> > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > at
> scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > fa
> > > > ul
> > > > tEventHandler.scala:100)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > sc
> > > > ala:72)
> > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > at
> > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > at
> > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:21)
> > > > > [ WARN] [main 2014-01-27 15:59:44,139] Failed to send producer
> > > > > request with correlation id 2 to broker 2 with data for
> > > > > partitions [mytopic,0]
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > >
> > >
> >
>
Re: Reg Exception in Kafka
Posted by Jun Rao <ju...@gmail.com>.
It seems what's registered in ZK (10.199.31.87 <http://10.199.31.87:9094/>)
is still the local ip, not the public one. Could you check the broker
registration in zk (
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper)
and see what's the host/port of the broker?
Thanks,
Jun
On Tue, Feb 4, 2014 at 1:54 AM, Balasubramanian Jayaraman (Contingent) <
balasubramanian.jayaraman@autodesk.com> wrote:
> I downloaded from the trunk and set up the properties
> host.name= <local IP>
> advertise.host.name=<public IP>
>
> Even after this change, I get the ConnectException. The details logs are
> given below. Is there any workaround for this?
>
> [ INFO] [main 2014-02-04 17:46:01,775] Disconnecting from
> 54.241.44.129:9094
> [DEBUG] [main 2014-02-04 17:46:01,780] Successfully fetched metadata for 1
> topic(s) Set(mytopic)
> [DEBUG] [main 2014-02-04 17:46:01,798] Getting broker partition info for
> topic mytopic
> [DEBUG] [main 2014-02-04 17:46:01,799] Partition [mytopic,0] has leader 3
> [DEBUG] [main 2014-02-04 17:46:01,807] Broker partitions registered for
> topic: mytopic are 0
> [DEBUG] [main 2014-02-04 17:46:01,820] Sending 1 messages with no
> compression to [mytopic,0]
> [DEBUG] [main 2014-02-04 17:46:01,833] Producer sending messages with
> correlation id 2 for topics [mytopic,0] to broker 3 on 10.199.31.87:9094
> [ERROR] [main 2014-02-04 17:46:22,850] Producer connection to
> 10.199.31.87:9094 unsuccessful
> java.net.ConnectException: Connection timed out: connect
> at sun.nio.ch.Net.connect0(Native Method)
> at sun.nio.ch.Net.connect(Net.java:465)
> at sun.nio.ch.Net.connect(Net.java:457)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:666)
> at kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> at kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> at
> kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161)
> at
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:68)
> at
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
> at
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> at
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
> at
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> at
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> at
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:254)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
> at
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> at
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> at
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
> at
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> at kafka.producer.Producer.send(Producer.scala:76)
> at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> at
> com.autodesk.kafka.test.utils.KafkaProducer.sendMessage(KafkaProducer.java:48)
> at
> com.autodesk.kafka.test.integration.KafkaProducerTest.testProducer(KafkaProducerTest.java:33)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at junit.framework.TestCase.runTest(TestCase.java:154)
> at junit.framework.TestCase.runBare(TestCase.java:127)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:118)
> at junit.framework.TestSuite.runTest(TestSuite.java:208)
> at junit.framework.TestSuite.run(TestSuite.java:203)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:131)
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
> [ WARN] [main 2014-02-04 17:46:22,862] Failed to send producer request
> with correlation id 2 to broker 3 with data for partitions [mytopic,0]
>
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Wednesday, January 29, 2014 11:12 PM
> To: users@kafka.apache.org
> Subject: Re: Reg Exception in Kafka
>
> Hmm, it's weird that EC2 only allows you to bind to local ip. Could some
> EC2 users here help out?
>
> Also, we recently added https://issues.apache.org/jira/browse/KAFKA-1092,
> which allows one to use a different ip for binding and connecting. You can
> see if this works for you. The patch is only in trunk though.
>
> Thanks,
>
> Jun
>
>
> On Tue, Jan 28, 2014 at 10:10 PM, Balasubramanian Jayaraman (Contingent) <
> balasubramanian.jayaraman@autodesk.com> wrote:
>
> > I don't think so. I forgot to include the ifconfig output. Actually
> > the public IP is not one of the IP configured in the Ethernet interfaces.
> > Only the Local IP is configured in eth0.
> > Is there any solution to this?
> >
> > Ifconfig O/P:
> >
> > eth0 Link encap:Ethernet HWaddr 22:00:0A:C7:1F:57
> > inet addr:10.X.X.X Bcast:10.199.31.127 Mask:255.255.255.192
> > inet6 addr: fe80::2000:aff:fec7:1f57/64 Scope:Link
> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
> > RX packets:83186 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:91285 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:1000
> > RX bytes:40233350 (38.3 MiB) TX bytes:15089154 (14.3 MiB)
> > Interrupt:25
> >
> > lo Link encap:Local Loopback
> > inet addr:127.0.0.1 Mask:255.0.0.0
> > inet6 addr: ::1/128 Scope:Host
> > UP LOOPBACK RUNNING MTU:16436 Metric:1
> > RX packets:1379711 errors:0 dropped:0 overruns:0 frame:0
> > TX packets:1379711 errors:0 dropped:0 overruns:0 carrier:0
> > collisions:0 txqueuelen:0
> > RX bytes:109133672 (104.0 MiB) TX bytes:109133672 (104.0
> > MiB)
> >
> > Thanks
> > Bala
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Wednesday, January 29, 2014 12:27 PM
> > To: users@kafka.apache.org
> > Subject: Re: Reg Exception in Kafka
> >
> > Could it be a port conflict?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Jan 28, 2014 at 5:20 PM, Balasubramanian Jayaraman
> > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> >
> > > Jun,
> > >
> > > Thanks for your help.
> > > I get the following exception :
> > > kafka.common.KafkaException: Socket server failed to bind to
> > > 54.241.44.129:9092: Cannot assign requested address.
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > at
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > at kafka.Kafka$.main(Kafka.scala:46)
> > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > java.net.BindException: Cannot assign requested address
> > > at sun.nio.ch.Net.bind0(Native Method)
> > > at sun.nio.ch.Net.bind(Net.java:174)
> > > at
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > ... 6 more
> > >
> > > The entire stack trace of the logs are placed below.
> > >
> > > [2014-01-29 01:18:23,136] INFO Verifying properties
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,176] INFO Property host.name is overridden to
> > > 54.241.44.129 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,177] INFO Property port is overridden to 9092
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,177] INFO Property socket.request.max.bytes is
> > > overridden to 104857600 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,177] INFO Property num.io.threads is overridden
> > > to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,178] INFO Property log.dirs is overridden to
> > > /tmp/kafka-logs-1 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,178] INFO Property log.cleanup.interval.mins is
> > > overridden to 1 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,178] INFO Property socket.send.buffer.bytes is
> > > overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,179] INFO Property log.flush.interval.ms is
> > > overridden to 1000 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,179] INFO Property zookeeper.connect is
> > > overridden to
> > > localhost:2181 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property broker.id is overridden to 1
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property log.retention.hours is
> > > overridden to 168 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property num.network.threads is
> > > overridden to 2 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,180] INFO Property socket.receive.buffer.bytes
> > > is overridden to 1048576 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,181] INFO Property
> > > zookeeper.connection.timeout.msis overridden to 1000000
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,181] INFO Property num.partitions is overridden
> > > to 2
> > > (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,181] INFO Property log.flush.interval.messages
> > > is overridden to 10000 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,182] INFO Property log.segment.bytes is
> > > overridden to
> > > 536870912 (kafka.utils.VerifiableProperties)
> > > [2014-01-29 01:18:23,198] INFO [Kafka Server 1], Starting
> > > (kafka.server.KafkaServer)
> > > [2014-01-29 01:18:23,248] INFO [Log Manager on Broker 1] Starting
> > > log cleaner every 60000 ms (kafka.log.LogManager)
> > > [2014-01-29 01:18:23,260] INFO [Log Manager on Broker 1] Starting
> > > log flusher every 3000 ms with the following overrides Map()
> > > (kafka.log.LogManager)
> > > [2014-01-29 01:18:23,330] FATAL Fatal error during KafkaServerStable
> > > startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> > > kafka.common.KafkaException: Socket server failed to bind to
> > > 54.241.44.129:9092: Cannot assign requested address.
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:188)
> > > at kafka.network.Acceptor.<init>(SocketServer.scala:134)
> > > at kafka.network.SocketServer.startup(SocketServer.scala:61)
> > > at kafka.server.KafkaServer.startup(KafkaServer.scala:77)
> > > at
> > >
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
> > > at kafka.Kafka$.main(Kafka.scala:46)
> > > at kafka.Kafka.main(Kafka.scala) Caused by:
> > > java.net.BindException: Cannot assign requested address
> > > at sun.nio.ch.Net.bind0(Native Method)
> > > at sun.nio.ch.Net.bind(Net.java:174)
> > > at
> > >
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:139)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:77)
> > > at
> > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:70)
> > > at
> > kafka.network.Acceptor.openServerSocket(SocketServer.scala:184)
> > > ... 6 more
> > > [2014-01-29 01:18:23,333] INFO [Kafka Server 1], Shutting down
> > > (kafka.server.KafkaServer)
> > > [2014-01-29 01:18:23,335] INFO [Socket Server on Broker 1], Shutting
> > > down
> > > (kafka.network.SocketServer)
> > > [2014-01-29 01:18:23,339] INFO [Socket Server on Broker 1], Shutdown
> > > completed (kafka.network.SocketServer)
> > > [2014-01-29 01:18:23,341] INFO Shutdown Kafka scheduler
> > > (kafka.utils.KafkaScheduler)
> > > [2014-01-29 01:18:23,360] INFO [Kafka Server 1], Shut down completed
> > > (kafka.server.KafkaServer)
> > > [2014-01-29 01:18:23,384] INFO [Kafka Server 1], Shutting down
> > > (kafka.server.KafkaServer)
> > >
> > > Regards
> > > Bala
> > >
> > > -----Original Message-----
> > > From: Jun Rao [mailto:junrao@gmail.com]
> > > Sent: Tuesday, January 28, 2014 11:30 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: Reg Exception in Kafka
> > >
> > > You should should use the public IP for host.name. What's the error
> > > you see during broker startup?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Tue, Jan 28, 2014 at 2:17 AM, Balasubramanian Jayaraman
> > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > >
> > > > I checked the faq. I did change the host.name in server properties.
> > > > After changing it I get ConnectException.
> > > >
> > > > The problem here is in EC2 have a different public IP address
> > > > (55.x.x.x) and the local IP address is (10.x.x.x).
> > > > I set the host.name property to the local IP address which is
> > > > 10.x.x.x. I think because of this there is a ConnectException.
> > > > When I set the host.name to the public ip address (55.x.x.x), I
> > > > cannot even start the broker.
> > > >
> > > > What should be the IP address that is to be given in
> host.nameproperty.
> > > >
> > > > Thanks
> > > > Bala
> > > >
> > > > -----Original Message-----
> > > > From: Jun Rao [mailto:junrao@gmail.com]
> > > > Sent: Tuesday, January 28, 2014 1:11 AM
> > > > To: users@kafka.apache.org
> > > > Subject: Re: Reg Exception in Kafka
> > > >
> > > > Have you looked at
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-OnEC2,wh
> > > > yc an 'tmyhigh-levelconsumersconnecttothebrokers
> > > > ?
> > > >
> > > > thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Mon, Jan 27, 2014 at 12:17 AM, Balasubramanian Jayaraman
> > > > (Contingent) < balasubramanian.jayaraman@autodesk.com> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I have a remote server (EC2) setup with Kafka cluster setup.
> > > > > There are
> > > > > 3 brokers each running in the port 9092,9093,9094. The zookeeper
> > > > > is running in the port 2181.
> > > > > When I send message to the brokers from my PC, I get an
> > > > > exception which is given below. I did a dump in the remote
> > > > > server, the request is received in the remote server.
> > > > > I am able to locally test the consumer/producer script present
> > > > > in the bin folder. What am I missing? Can you kindly help me in
> > > > > this
> > > error?
> > > > > Any help will be highly grateful.
> > > > >
> > > > > [ INFO] [main 2014-01-27 16:06:50,083] Verifying properties [
> > > > > INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > metadata.broker.list is overridden to
> > > > > 54.241.44.129:9092,54.241.44.129:9093,54.241.44.129:9094
> > > > > [ INFO] [main 2014-01-27 16:06:50,108] Property
> > > > > request.required.acks is overridden to 1 [ INFO] [main
> > > > > 2014-01-27 16:06:50,108] Property key.serializer.class is
> > > > > overridden to kafka.serializer.StringEncoder [ INFO] [main
> > > > > 2014-01-27 16:06:50,108] Property serializer.class is overridden
> > > > > to kafka.utils.EncryptEncoder [ INFO] [main 2014-01-27
> > > > > 16:06:50,154]
> > > > > send: encrypted - Message_1 [DEBUG] [main 2014-01-27
> > > > > 16:06:50,298] Handling 1 events [ INFO] [main
> > > > > 2014-01-27 15:59:43,540] Fetching metadata from broker
> > > > > id:0,host:54.241.44.129,port:9093 with correlation id 0 for 1
> > > > > topic(s)
> > > > > Set(mytopic)
> > > > > [DEBUG] [main 2014-01-27 15:59:43,737] Created socket with
> > > > > SO_TIMEOUT =
> > > > > 10000 (requested 10000), SO_RCVBUF = 8192 (requested -1),
> > > > > SO_SNDBUF =
> > > > > 102400 (requested 102400).
> > > > > [ INFO] [main 2014-01-27 15:59:43,738] Connected to
> > > > > 54.241.44.129:9093for producing [ INFO] [main 2014-01-27
> > > > > 15:59:44,018] Disconnecting from
> > > > > 54.241.44.129:9093
> > > > > [DEBUG] [main 2014-01-27 15:59:44,025] Successfully fetched
> > > > > metadata for 1
> > > > > topic(s) Set(mytopic)
> > > > > [DEBUG] [main 2014-01-27 15:59:44,058] Getting broker partition
> > > > > info for topic mytopic [DEBUG] [main 2014-01-27 15:59:44,060]
> > > > > Partition [mytopic,0] has leader 2 [DEBUG] [main 2014-01-27
> > > > > 15:59:44,072] Broker partitions registered for
> > > > > topic: mytopic are 0
> > > > > [DEBUG] [main 2014-01-27 15:59:44,091] Sending 1 messages with
> > > > > no compression to [mytopic,0] [DEBUG] [main 2014-01-27
> > > > > 15:59:44,109] Producer sending messages with correlation id 2
> > > > > for topics [mytopic,0] to broker 2 on
> > > > > ip-10-199-31-87.us-west-1.compute.internal:9093
> > > > > [ERROR] [main 2014-01-27 15:59:44,129] Producer connection to
> > > > > ip-10-199-31-87.us-west-1.compute.internal:9093 unsuccessful
> > > > > java.nio.channels.UnresolvedAddressException
> > > > > at sun.nio.ch.Net.checkAddress(Net.java:127)
> > > > > at
> > > > sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:640)
> > > > > at
> > > kafka.network.BlockingChannel.connect(BlockingChannel.scala:57)
> > > > > at
> kafka.producer.SyncProducer.connect(SyncProducer.scala:146)
> > > > > at
> > > > >
> > kafka.producer.SyncProducer.getOrMakeConnection(SyncProducer.scala:161
> > )
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(Sy
> > > > nc
> > > > Pr
> > > > oducer.scala:68)
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > ap
> > > > ply$mcV$sp(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > ap
> > > > ply(SyncProducer.scala:102)
> > > > > at
> > > > >
> > > > kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.
> > > > ap
> > > > ply(SyncProducer.scala:102)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at
> > > > >
> > > >
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.
> > > > scala:101)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > :1
> > > 01)
> > > > > at
> > > > >
> > > kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala
> > > :1
> > > 01)
> > > > > at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler.kafka$producer$async$Defa
> > > > ul
> > > > tE
> > > > ventHandler$$send(DefaultEventHandler.scala:254)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > ed
> > > > Da
> > > > ta$1.apply(DefaultEventHandler.scala:106)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializ
> > > > ed
> > > > Da
> > > > ta$1.apply(DefaultEventHandler.scala:100)
> > > > > at
> > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > sc
> > > > al
> > > > a:80)
> > > > > at
> > > > >
> > > > scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.
> > > > sc
> > > > al
> > > > a:80)
> > > > > at
> scala.collection.Iterator$class.foreach(Iterator.scala:631)
> > > > > at
> > > > >
> > scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161
> > )
> > > > > at
> > > > >
> > > >
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:
> > > > 194)
> > > > > at
> > > scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> > > > > at
> scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler.dispatchSerializedData(De
> > > > fa
> > > > ul
> > > > tEventHandler.scala:100)
> > > > > at
> > > > >
> > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.
> > > > sc
> > > > ala:72)
> > > > > at kafka.producer.Producer.send(Producer.scala:76)
> > > > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > > > at
> > > > > kafka.application.KafkaProducer.sendMessage(KafkaProducer.java:39)
> > > > > at
> > > > > kafka.test.KafkaProducerTest.main(KafkaProducerTest.java:21)
> > > > > [ WARN] [main 2014-01-27 15:59:44,139] Failed to send producer
> > > > > request with correlation id 2 to broker 2 with data for
> > > > > partitions [mytopic,0]
> > > > >
> > > > > Thanks
> > > > > Bala
> > > > >
> > > >
> > >
> >
>