[jira] [Created] (AURORA-1972) Flapping Excutor HealthChecker Test

["Stephan Erb (JIRA)" <ji...@apache.org>]

Stephan Erb created AURORA-1972:
-----------------------------------

             Summary: Flapping Excutor HealthChecker Test
                 Key: AURORA-1972
                 URL: https://issues.apache.org/jira/browse/AURORA-1972
             Project: Aurora
          Issue Type: Bug
          Components: Executor
            Reporter: Stephan Erb


We currently have a set of flapping HealthChecking test that prevent our builds from passing (e.g. see https://reviews.apache.org/r/65565/).

{code}
                     ==================== FAILURES ====================
                      TestThreadedHealthCheckerWithDefaults.test_run_unhealthy_after_callback

                     self = <apache.aurora.executor.common.test_health_checker.TestThreadedHealthCheckerWithDefaults testMethod=test_run_unhealthy_after_callback>
                     mock_sleep = <MagicMock name='sleep' spec='builtin_function_or_method' id='4598488912'>

                         @mock.patch('apache.aurora.executor.common.health_checker.time.sleep', spec=time.sleep)
                         def test_run_unhealthy_after_callback(self, mock_sleep):
                           mock_sleep.return_value = None
                           health_status = [(True, None), (True, None), (False, 'failure-4'), (False, 'failure-5')]
                           self.health.side_effect = lambda: health_status.pop(0)
                           mock_is_set = mock.Mock(spec=threading._Event.is_set)
                           liveness = [False, False, False, False, True]
                           mock_is_set.side_effect = lambda: liveness.pop(0)
                           self.health_checker.threaded_health_checker.dead.is_set = mock_is_set
                           self.health_checker.threaded_health_checker.run()
                     >     assert mock_sleep.call_count == 4
                     E     AssertionError: assert 9403 == 4
                     E      +  where 9403 = <MagicMock name='sleep' spec='builtin_function_or_method' id='4598488912'>.call_count

                     .pants.d/pyprep/sources/365105c9a0472d6a1d7576426d316fe2aa7dcc77/apache/aurora/executor/common/test_health_checker.py:1292: AssertionError
{code}
Please notice the huge difference between actual and expected calls.



This works as expected:
{code}
./pants --cache-ignore --no-test-pytest-fast test.pytest  src/test/python::
{code}

This triggers the problem with traces as the one posted above:
{code}
./pants --cache-ignore --no-test-pytest-fast test.pytest  src/test/python::
{code}

The flapping seems to be dependent on how pants executes the tests. This seems to have a side effect on how {{time.sleep}} mocking is performed.  

The relevant [test class|https://github.com/apache/aurora/blob/c85bffdd6f68312261697eee868d57069adda434/src/test/python/apache/aurora/executor/common/test_health_checker.py#L1033] and the [failing tests|https://github.com/apache/aurora/blob/c85bffdd6f68312261697eee868d57069adda434/src/test/python/apache/aurora/executor/common/test_health_checker.py#L1265-L1280] in particular are somewhat low-quality as they make heavy use of mocking.

How do we want to proceed here? 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)