[jira] [Resolved] (IMPALA-5708) Test failure with invalid GetExecSummary; potential coord. race

["Thomas Tauber-Marshall (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Tauber-Marshall resolved IMPALA-5708.
--------------------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

commit 6757b6235c68f3886e28ecda8fc6598305717d2e
Author: Thomas Tauber-Marshall <tm...@cloudera.com>
Date:   Wed Aug 9 09:17:37 2017 -0700

    IMPALA-5708: Test failure with invalid exec summary
    
    For some queries, the exec summary will not be completely filled
    in even if the query is FINISHED. In particular, the exec_stats field
    may not be set. This was causing an error in our test code that
    converts the exec summary to a more usable format.
    
    The situation is essentially deterministic for some queries, but
    it was being hidden by testing code that caught the error and
    discarded it in most situations, leading to flaky tests.
    
    This patch removes the 'try' that was hiding the error and makes
    the code check for the presence of exec_stats and handle it rather
    than generating an error.
    
    I filed IMPALA-5783 for followup work to be more rigorous about
    when the exec summary should and shouldn't be fully present.
    
    Testing:
    - Ran the affected tests in a loop and they are no longer flaky.
    
    Change-Id: Id52ac62da2b01f9e163e97cbe4590f8db6b663d2
    Reviewed-on: http://gerrit.cloudera.org:8080/7627
    Tested-by: Impala Public Jenkins
    Reviewed-by: Thomas Tauber-Marshall <tm...@cloudera.com>

> Test failure with invalid GetExecSummary; potential coord. race
> ---------------------------------------------------------------
>
>                 Key: IMPALA-5708
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5708
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.10.0
>            Reporter: Matthew Jacobs
>            Assignee: Thomas Tauber-Marshall
>            Priority: Critical
>              Labels: flaky-test
>             Fix For: Impala 2.10.0
>
>         Attachments: impalad.log, profile.b64
>
>
> A jenkins test job failed in 
> {code}
> query_test.test_queries.TestHdfsQueries.test_union[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: seq/bzip/block] (from pytest)
> {code}
> with the following stack trace in the pytest:
> {code}
> query_test/test_queries.py:90: in test_union
>     result = self.execute_query(query_string, vector.get_value('exec_option'))
> common/impala_test_suite.py:506: in wrapper
>     return function(*args, **kwargs)
> common/impala_test_suite.py:531: in execute_query
>     return self.__execute_query(self.client, query, query_options)
> common/impala_test_suite.py:598: in __execute_query
>     return impalad_client.execute(query, user=user)
> common/impala_connection.py:160: in execute
>     return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:194: in execute
>     result.exec_summary = self.get_exec_summary(handle)
> beeswax/impala_beeswax.py:213: in get_exec_summary
>     self.__build_summary_table(summary, 0, False, 0, False, output)
> beeswax/impala_beeswax.py:252: in __build_summary_table
>     for stats in node.exec_stats:
> E   TypeError: 'NoneType' object is not iterable
> {code}
> It looks to me like the returned ExecSummary is wrong or invalid. Nothing very obvious in the logs (attached a snip that I believe is the relevant portion), and I found the profile that was written to the profile log (attached as well). I suspect something may be wrong with the profile because CM doesn't seem to decode it: when I loaded the encoded profile in the CM debug tool "impalaProfileDecoder" (e.g. http://cm-host:7180/cmf/debug/impalaProfileDecoder ), there was no profile details, and exporting as text resulted in a file that just said "Unknown profile.". A next step would be to try to decode the profile in some other way to check validity.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)