You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Shanthoosh Venkataraman (JIRA)" <ji...@apache.org> on 2018/02/13 01:24:00 UTC
[jira] [Updated] (SAMZA-1586) Kafka broker setup problem in
standalone integration tests.
[ https://issues.apache.org/jira/browse/SAMZA-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shanthoosh Venkataraman updated SAMZA-1586:
-------------------------------------------
Description:
We've observed in standalone integration tests that embedded kafka broker was unreachable and kafka NetworkClient spams the logs with the following exception messages.
{code:java}
99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to java.nio.channels.ClosedChannelException. Releasing ownership of all partitions, and restarting consumer. Turn on debugging to get a full stack trace.
99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,0]
99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,1]
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,1] -> 20)
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,0] -> 20)
99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
{code}
Reading/writing into kafka within the integration test fails because of the above problem.
Majority of the standalone integration tests that fail with junit test timeout were stuck in the infinite retry loop of BrokerProxy I/O thread which polls messages from the embedded kafka broker.
This requires further investigation.
was:
We've observed in standalone integration tests that embedded kafka broker was unreachable and kafka NetworkClient spams the logs with the following exception messages.
{code:java}
99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to java.nio.channels.ClosedChannelException. Releasing ownership of all partitions, and restarting consumer. Turn on debugging to get a full stack trace.
99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,0]
99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,1]
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,1] -> 20)
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,0] -> 20)
99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
{code}
Reading/writing into kafka within the integration test fails because of the above problem.
Majority of the integration tests that fails with junit test timeout are due to the following:
were stuck in the infinite retry loop of BrokerProxy I/O thread polling messages from the embedded kafka broker(or trying to produce messages and failing ).
This requires further investigation.
> Kafka broker setup problem in standalone integration tests.
> -----------------------------------------------------------
>
> Key: SAMZA-1586
> URL: https://issues.apache.org/jira/browse/SAMZA-1586
> Project: Samza
> Issue Type: Bug
> Reporter: Shanthoosh Venkataraman
> Assignee: Shanthoosh Venkataraman
> Priority: Minor
>
> We've observed in standalone integration tests that embedded kafka broker was unreachable and kafka NetworkClient spams the logs with the following exception messages.
> {code:java}
> 99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to java.nio.channels.ClosedChannelException. Releasing ownership of all partitions, and restarting consumer. Turn on debugging to get a full stack trace.
> 99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,0]
> 99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,1]
> 99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,1] -> 20)
> 99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,0] -> 20)
> 99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
> 100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> 100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
> {code}
>
> Reading/writing into kafka within the integration test fails because of the above problem.
> Majority of the standalone integration tests that fail with junit test timeout were stuck in the infinite retry loop of BrokerProxy I/O thread which polls messages from the embedded kafka broker.
> This requires further investigation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)