[jira] [Created] (KAFKA-5608) System test failure due to timeout starting Jmx tool

["Jason Gustafson (JIRA)" <ji...@apache.org>]

Jason Gustafson created KAFKA-5608:
--------------------------------------

             Summary: System test failure due to timeout starting Jmx tool
                 Key: KAFKA-5608
                 URL: https://issues.apache.org/jira/browse/KAFKA-5608
             Project: Kafka
          Issue Type: Bug
            Reporter: Jason Gustafson
            Assignee: Ewen Cheslack-Postava


Began seeing this in some failing system tests:
{code}
[INFO  - 2017-07-18 14:25:55,375 - background_thread - _protected_worker - lineno:39]: Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/services/background_thread.py", line 35, in _protected_worker
    self._worker(idx, node)
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/services/console_consumer.py", line 261, in _worker
    self.start_jmx_tool(idx, node)
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/services/monitor/jmx.py", line 73, in start_jmx_tool
    wait_until(lambda: self._jmx_has_output(node), timeout_sec=10, backoff_sec=.5, err_msg="%s: Jmx tool took too long to start" % node.account)
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/utils/util.py", line 36, in wait_until
    raise TimeoutError(err_msg)
TimeoutError: ubuntu@worker7: Jmx tool took too long to start
{code}

This is immediately followed by a consumer timeout in the failing cases:
{code}
[INFO  - 2017-07-18 14:26:46,907 - runner_client - log - lineno:221]: RunnerClient: kafkatest.tests.core.security_rolling_upgrade_test.TestSecurityRollingUpgrade.test_rolling_upgrade_phase_two.broker_protocol=SASL_SSL.client_protocol=SASL_SSL: FAIL: Consumer failed to consume messages for 60s.
Traceback (most recent call last):
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", line 123, in run
    data = self.run_test()
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py", line 176, in run_test
    return self.test_context.function(self.test)
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/mark/_mark.py", line 321, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py", line 148, in test_rolling_upgrade_phase_two
    self.run_produce_consume_validate(self.roll_in_secured_settings, client_protocol, broker_protocol)
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/tests/produce_consume_validate.py", line 106, in run_produce_consume_validate
    self.start_producer_and_consumer()
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/tests/kafkatest/tests/produce_consume_validate.py", line 79, in start_producer_and_consumer
    self.consumer_start_timeout_sec)
  File "/home/jenkins/workspace/system-test-kafka-0.11.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/utils/util.py", line 36, in wait_until
    raise TimeoutError(err_msg)
TimeoutError: Consumer failed to consume messages for 60s.
{code}

There does not appear to be anything wrong with the consumer in the logs, so the timeout seems to be caused by the Jmx tool timeout.

Possibly due to https://github.com/apache/kafka/pull/3447/files?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)