[jira] [Commented] (KAFKA-1494) Failed to send messages after 3 tries.

["Josh Rosen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154518#comment-14154518 ] 

Josh Rosen commented on KAFKA-1494:
-----------------------------------

I'm able to reproduce a similar problem using Kafka 0.8 with a single broker running inside of Docker (using boot2docker) on OS X.  In this setup, I run a single-node ZooKeeper process in its own Docker container and configure the Kafka broker to use it.  The broker is configured to bind to the Docker container's IP on the VirtualBox network.  Using {{sudo route \-n add 172.17.0.0/16 `boot2docker ip`}}, I've configured my Mac so that programs running outside of Docker can access network services running inside of containers using the containers' own IP addresses (so we use the same IPs for host-container and container-container communication).  I have some Scala code which uses Kafka's ZkUtils to retrieve information on the Kafka cluster state and kafka.producer.Producer to push data into Kafka.

When I attempt to run this, I see errors very similar to the ones reported in this ticket:

{code}
14/10/01 00:22:54 INFO client.ClientUtils$: Fetching metadata from broker id:0,host:172.17.0.187,port:12345 with correlation id 11 for 1 topic(s) Set(test-topic)
14/10/01 00:22:54 INFO producer.SyncProducer: Connected to 172.17.0.187:12345 for producing
14/10/01 00:22:54 INFO producer.SyncProducer: Disconnecting from 172.17.0.187:12345
14/10/01 00:22:54 WARN producer.BrokerPartitionInfo: Error while fetching metadata [{TopicMetadata for topic test-topic ->
No partition metadata for topic test-topic due to kafka.common.LeaderNotAvailableException}] for topic [test-topic]: class kafka.common.LeaderNotAvailableException
{code}

I've found messages on the mailing list that describe this LeaderNotAvailableException; however, I don't think that any of the root causes reported there apply in my case:


- My producer can connect to the broker (I know this because I set `client.id` in my producer configuration and see matching entries in the broker's logs).
- My broker can connect to ZooKeeper (I manually inspected ZooKeeper's contents using zookeeper-shell).
- I didn't attempt to delete any topics.
- ZooKeeper starts with a completely fresh state; leftover state from earlier attempts / experiments isn't an issue here.
- I don't think there are any race-conditions in my client code; I added long (20-30 second) Thread.sleep() calls in my program so that we wait for leader election / metadata propagation to occur after creating the new topic.


I've noticed that ZkUtils.getReplicasForPartition lists the expected replica for the partition, although ZkUtils.getLeaderForPartition returns None, even after multiple attempts / long delays.

This test is easy for me to re-run; the entire setup / teardown of the containers takes less than a minute.  Therefore, it's very easy for me to experiment with different configurations / settings.  I can provide any logs, configuration files, etc. that would be helpful when debugging this; just let me know what you need.

> Failed to send messages after 3 tries.
> --------------------------------------
>
>                 Key: KAFKA-1494
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1494
>             Project: Kafka
>          Issue Type: Bug
>          Components: controller, core
>    Affects Versions: 0.8.1.1
>         Environment: Mac OS 
>            Reporter: darion yaphets
>            Assignee: Neha Narkhede
>
> I use default server & zookeeper config to start-up zookeeper server and kafka broker on my machine to test custom message which based on proto buffer . I write a client to send protobuf-message to kafka broker and source code as following :
>                 Properties properties = new Properties();
> 		properties.put("serializer.class", "java_example.ProtoBufMessage");
> 		properties.put("metadata.broker.list", "localhost:9092");
> 		ProducerConfig config = new ProducerConfig(properties);
> 		testBuf buffer = testBuf.newBuilder().setID(0)
> 				.setUrl("darion.yaphet.org").build();
> 		Producer<String, testBuf> producer = new Producer<String, testBuf>(
> 				config);
> 		producer.send(new KeyedMessage<String, testBuf>("protobuffer", buffer));
> client debug log report a exception:
> [FileSystemMoniter] INFO [main] kafka.utils.Logging$class.info(68) | Disconnecting from localhost:9092
> [FileSystemMoniter] DEBUG [main] kafka.utils.Logging$class.debug(52) | Successfully fetched metadata for 1 topic(s) Set(protobuffer)
> [FileSystemMoniter] WARN [main] kafka.utils.Logging$class.warn(83) | Error while fetching metadata [{TopicMetadata for topic protobuffer -> 
> No partition metadata for topic protobuffer due to kafka.common.LeaderNotAvailableException}] for topic [protobuffer]: class kafka.common.LeaderNotAvailableException 
> [FileSystemMoniter] ERROR [main] kafka.utils.Logging$class.error(97) | Failed to send requests for topics protobuffer with correlation ids in [0,8]
> Exception in thread "main" kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries.
> 	at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> 	at kafka.producer.Producer.send(Producer.scala:76)
> 	at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> 	at java_example.ProducerExamples.main(ProducerExamples.java:26)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)