[jira] [Created] (IMPALA-6351) TestValidateMetrics failing for rhel tests

["Vuk Ercegovac (JIRA)" <ji...@apache.org>]

Vuk Ercegovac created IMPALA-6351:
-------------------------------------

             Summary: TestValidateMetrics failing for rhel tests
                 Key: IMPALA-6351
                 URL: https://issues.apache.org/jira/browse/IMPALA-6351
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
    Affects Versions: Impala 2.10.0
            Reporter: Vuk Ercegovac


On a recent rhel exhaustive test (kicked off on 12/21):

verifiers/test_verify_metrics.py:34: in test_metrics_are_zero
    verifier.verify_metrics_are_zero()
verifiers/metric_verifier.py:47: in verify_metrics_are_zero
    self.wait_for_metric(metric, 0, timeout)
verifiers/metric_verifier.py:62: in wait_for_metric
    self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
common/impala_service.py:89: in wait_for_metric_value
    (metric_name, expected_value, timeout)
E   AssertionError: Metric value impala-server.mem-pool.total-bytes did not reach value 0 in 60s

Here is the metric that was obtained:

...
MainThread: Getting metric: impala-server.mem-pool.total-bytes from localhost:25000
MainThread: Waiting for metric value 'impala-server.mem-pool.total-bytes'=0. Current value: 3290112
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.mem-pool.total-bytes from localhost:25000
MainThread: Waiting for metric value 'impala-server.mem-pool.total-bytes'=0. Current value: 3290112
MainThread: Sleeping 1s before next retry.

Similarly, the number of unused buffers did not reach the expected value:
verifiers/test_verify_metrics.py:39: in test_num_unused_buffers
    verifier.verify_num_unused_buffers()
verifiers/metric_verifier.py:59: in verify_num_unused_buffers
    timeout)
verifiers/metric_verifier.py:62: in wait_for_metric
    self.impalad_service.wait_for_metric_value(metric_name, expected_value, timeout)
common/impala_service.py:89: in wait_for_metric_value
    (metric_name, expected_value, timeout)
E   AssertionError: Metric value impala-server.io-mgr.num-unused-buffers did not reach value 204 in 60s

And the value that was observed:
....
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.
MainThread: Getting metric: impala-server.io-mgr.num-unused-buffers from localhost:25000
MainThread: Waiting for metric value 'impala-server.io-mgr.num-unused-buffers'=204. Current value: 203
MainThread: Sleeping 1s before next retry.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)