[jira] [Resolved] (IMPALA-6241) Timeout in TestAdmissionControllerStress.test_mem_limit under ASAN

["Tim Armstrong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-6241.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.11.0


  IMPALA-6241: timeout in admission control test under ASAN

The fix for IMPALA-6241 is to increase the timeout for all slow builds.

While testing that fix, I discovered that the ASAN build detection logic
was failing silently, resulting in it assuming that it was testing a
DEBUG build. The error was:

  Unexpected DW_AT_name in first CU:
  /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-ubuntu-16-04/toolchain/source/llvm/llvm-3.9.1.src/projects/compiler-rt/lib/asan/asan_preinit.cc;
  choosing DEBUG

The fix for that issue is to remove the build type detection heuristic
and instead just write a file with the build type as part of the build process.

Testing:
Before this change I was able to reproduce locally every 5-10 test
iterations. After this change I haven't seen it reproduce.

Change-Id: Ia4ed949cac99b9925f72e19e4adaa2ead370b536
Reviewed-on: http://gerrit.cloudera.org:8080/8652
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins


> Timeout in TestAdmissionControllerStress.test_mem_limit under ASAN
> ------------------------------------------------------------------
>
>                 Key: IMPALA-6241
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6241
>             Project: IMPALA
>          Issue Type: Bug
>    Affects Versions: Impala 2.11.0
>            Reporter: Tianyi Wang
>            Assignee: Tim Armstrong
>            Priority: Blocker
>              Labels: broken-build
>             Fix For: Impala 2.11.0
>
>
> In an ASAN build, test_mem_limit times out in end_admitted_queries(). It's probably related to IMPALA-6227
> {noformat}
>     def end_admitted_queries(self, num_queries):
>       """
>         Requests each admitted query to end its query.
>         """
>       assert len(self.executing_threads) >= num_queries
>       LOG.debug("Requesting {0} clients to end queries".format(num_queries))
>     
>       # Request admitted clients to end their queries
>       current_executing_queries = []
>       for i in xrange(num_queries):
>         # pop() is thread-safe, it's OK if another thread is appending concurrently.
>         thread = self.executing_threads.pop(0)
>         LOG.debug("Cancelling query %s", thread.query_num)
>         assert thread.query_state == 'ADMITTED'
>         current_executing_queries.append(thread)
>         thread.query_state = 'REQUEST_QUERY_END'
>     
>       # Wait for the queries to end
>       start_time = time()
>       while True:
>         all_done = True
>         for thread in self.all_threads:
>           if thread.query_state == 'REQUEST_QUERY_END':
>             all_done = False
>         if all_done:
>           break
> >       assert (time() - start_time < STRESS_TIMEOUT),\
>           "Timed out waiting %s seconds for query end" % (STRESS_TIMEOUT,)
> E       AssertionError: Timed out waiting 60 seconds for query end
> E       assert (1511608615.681211 - 1511608555.6205289) < 60
> E        +  where 1511608615.681211 = time()
> custom_cluster/test_admission_controller.py:623: AssertionError
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)