You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "Shanthoosh Venkataraman (JIRA)" <ji...@apache.org> on 2018/02/13 01:23:00 UTC
[jira] [Created] (SAMZA-1586) Kafka broker setup problem in
standalone integration tests.
Shanthoosh Venkataraman created SAMZA-1586:
----------------------------------------------
Summary: Kafka broker setup problem in standalone integration tests.
Key: SAMZA-1586
URL: https://issues.apache.org/jira/browse/SAMZA-1586
Project: Samza
Issue Type: Bug
Reporter: Shanthoosh Venkataraman
Assignee: Shanthoosh Venkataraman
We've observed in standalone integration tests that embedded kafka broker was unreachable and kafka NetworkClient spams the logs with the following exception messages.
{code:java}
99097 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.BrokerProxy - Restarting consumer due to java.nio.channels.ClosedChannelException. Releasing ownership of all partitions, and restarting consumer. Turn on debugging to get a full stack trace.
99098 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,0]
99099 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Abdicating for [test_ZKS_numbers1,1]
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,1] -> 20)
99100 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] INFO org.apache.samza.system.kafka.KafkaSystemConsumer - Refreshing brokers for: Map([test_ZKS_numbers1,0] -> 20)
99191 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99198 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99204 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99298 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99300 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99351 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99401 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99404 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99406 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99452 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99455 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99505 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99508 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99556 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99606 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99608 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
99609 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99659 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99712 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99713 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99764 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99765 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99816 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99866 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99919 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99920 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
99972 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100009 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,0]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100011 [SAMZA-BROKER-PROXY-BrokerProxy thread pointed at localhost:36243 for client samza_consumer-test_job-1] WARN org.apache.samza.system.kafka.KafkaSystemConsumer - While refreshing brokers for [test_ZKS_numbers1,1]: kafka.common.KafkaException: fetching topic metadata for topics [Set(test_ZKS_numbers1)] from broker [ArrayBuffer(BrokerEndPoint(0,localhost,36243))] failed. Retrying.
100021 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100023 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100074 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100123 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100176 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100178 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100227 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100230 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100332 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100384 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100436 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100486 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100487 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100539 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100591 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100592 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100644 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100694 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100695 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100747 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
100796 [kafka-producer-network-thread | samza_producer-test_job-1] WARN org.apache.kafka.clients.NetworkClient - Connection to node 0 could not be established. Broker may not be available.
{code}
Reading/writing into kafka within the integration test fails because of the above problem.
Majority of the integration tests that fails with junit test timeout are due to the following:
were stuck in the infinite retry loop of BrokerProxy I/O thread polling messages from the embedded kafka broker(or trying to produce messages and failing ).
This requires further investigation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)