[jira] [Comment Edited] (IMPALA-6984) Coordinator should cancel backends when returning EOS

["Michael Ho (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-6984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692357#comment-16692357 ] 

Michael Ho edited comment on IMPALA-6984 at 11/19/18 10:19 PM:
---------------------------------------------------------------

Now that IMPALA-4063 is fixed, we should have a guarantee that the a final profile will be sent even when the fragment instances are cancelled. This is guaranteed as cancellation won't have any effect until all fragment instances on a backend have finished calling {{FIS:Prepare()}}  so the query state thread will definitely send one report upon completion of all fragment instances.

However, the logic in {{Coordinator::BackendState::Done()}} is preventing the final profile from being applied as {{status_}} is already set to {{CANCELLED}}. It may need to be fixed up in order for the coordinator to cleanly cancel the fragment instances on EOS.
{noformat}
inline bool Coordinator::BackendState::IsDone() const {
  return num_remaining_instances_ == 0 || !status_.ok();
}
{noformat}


was (Author: kwho):
Now that IMPALA-4063 is fixed, we should have a guarantee that the a final profile will be sent even when the fragment instances are cancelled. This is guaranteed as cancellation won't have any effect until the FIS on a backend has passed the preparation stage so the query state thread will definitely send one report upon completion of all fragment instancse.

However, the logic in {{Coordinator::BackendState::Done()}} is preventing the final profile from being applied as {{status_}} is already set to {{CANCELLED}}. It may need to be fixed up in order for the coordinator to cleanly cancel the fragment instances on EOS.
{noformat}
inline bool Coordinator::BackendState::IsDone() const {
  return num_remaining_instances_ == 0 || !status_.ok();
}
{noformat}

> Coordinator should cancel backends when returning EOS
> -----------------------------------------------------
>
>                 Key: IMPALA-6984
>                 URL: https://issues.apache.org/jira/browse/IMPALA-6984
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 3.0
>            Reporter: Dan Hecht
>            Priority: Major
>              Labels: query-lifecycle
>
> Currently, the Coordinator waits for backends rather than proactively cancelling them in the case of hitting EOS. There's a tangled mess that makes it tricky to proactively cancel the backends related to how {{Coordinator::ComputeQuerySummary()}} works – we can't update the summary until the profiles are no longer changing (which also makes sense given that we want the exec summary to be consistent with the final profile).  But we current tie together the FIS status and the profile, and cancellation of backends causes the FIS to return CANCELLED, which then means that the remaining FIS on that backend won't produce a final profile.
> With the rework of the protocol for IMPALA-2990 we should make it possible to sort this out such that a final profile can be requested regardless of how a FIS ends execution.
> This also relates to IMPALA-5783.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org