[jira] [Created] (IMPALA-9295) RPC failures don't always trigger a blacklist

["Sahil Takiar (Jira)" <ji...@apache.org>]

Sahil Takiar created IMPALA-9295:
------------------------------------

             Summary: RPC failures don't always trigger a blacklist
                 Key: IMPALA-9295
                 URL: https://issues.apache.org/jira/browse/IMPALA-9295
             Project: IMPALA
          Issue Type: Bug
          Components: Backend
            Reporter: Sahil Takiar
            Assignee: Sahil Takiar


There is a race condition in IMPALA-9137. It is possible for the aux_error_info and the failure status to arrive in separate exec status reports.

IMPALA-9137 added AuxErrorInfoPB to FragmentInstanceExecStatusPB (contains per-fragment info for a ReportExecStatusRequestPB). The idea is that if a query fails, the Coordinator would use the AuxErrorInfoPB to potentially blacklist any nodes that caused the failure. The Coordinator only looks for AuxErrorInfoPB if the query has failed (e.g. if ReportExecStatusRequestPB::overall_status is set to an error).

The issue is that is is possible that the AuxErrorInfoPB is set even though overall_status == OK. There is a race condition on the executor side where the setting of the aux_error_info and and overall_status is not synchronized. So if a fragment fails due to an RPC error, it is possible for report number "x" to include the aux_error_info with overall_status == OK, and report number "x + 1" to include no aux_error_info with overall_status == [some-RPC-failure-message].

Report num "x +1" won't include the aux_error_info since the fragment has finished and its last FragmentInstanceExecStatusPB was in report num "x".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)