[jira] [Created] (IMPALA-6025) Improve hang diagnostics

["Michael Ho (JIRA)" <ji...@apache.org>]

Michael Ho created IMPALA-6025:
----------------------------------

             Summary: Improve hang diagnostics
                 Key: IMPALA-6025
                 URL: https://issues.apache.org/jira/browse/IMPALA-6025
             Project: IMPALA
          Issue Type: Improvement
          Components: Backend, Distributed Exec
    Affects Versions: Impala 2.9.0
            Reporter: Michael Ho


In the past, users of Impalad had a hard time getting diagnostics information when a query is hung. Usually, that involves a rather manual process of determining the fragment instances which aren't making progress and generating stack trace or core from that Impalad and looking into it under a debugger. Given the thousand of threads running when multiple queries are active, it's quite time consuming for diagnostics.

This JIRA aims to track the improvement ideas which we can implement to alleviate the stress with debugging this kind of issue. Some ideas include:

- implement a diagnostic button (analogous to the cancellation button in the UI) to dump diagnostics information (e.g. threads' backtraces, executor nodes' internals, states of data stream sender and receivers, lock information (e.g. holder's pid) ) for fragment instances on some or all hosts of a query.

-  have a watch dog to dump backtraces on threads which aren't making progress for a while. This probably doesn't apply to all threads (e.g. idle threads shouldn't trigger any alert).

- A fragment instance can appear to be not making progress because its parent operator / fragment may be hung (e.g.the probe side of a join will not be able to make much progress until the build side is done and the build side itself could be another chain of joins). It'd be much easier to resolve this dependency chain programmatically to find the root of the cascade of delay.

Please feel free to add more ideas to this JIRA.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)