You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Bikramjeet Vig (Jira)" <ji...@apache.org> on 2021/08/17 20:00:04 UTC
[jira] [Updated] (IMPALA-10866) Ensure consistency between failure
detection and registration/Ack of a coordinator by the admission service
[ https://issues.apache.org/jira/browse/IMPALA-10866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bikramjeet Vig updated IMPALA-10866:
------------------------------------
Description:
Ensure consistency between failure detection and registration/Ack of a coordinator by the admission service.
Currently admission service utilizes the statestore membership updates to detect a coordinator going down but it still services RPCs from that coordinator if it is still up and able to contact the admission service.
Using the current mechanisms of statestore updates(IMPALA-10594), admission heartbeats(IMPALA-10590, IMPALA-10720) and coordinator registration(IMPALA-9976) ensure that consistency is maintained between these mechanism.
A possible implementation is:
- Use statestore as the only source of truth.
** Consistency: Only allow a coord to register if it is registered with the statestore
** Atomicity: If the statestore update signals that a coord is down, remove all its state (running and queued queries) before you allow it to register again
OR
Eventual consistency: We remove queries between subsequent statestore updates and if the coord comes back up and sends the full admission state, we can update the state of that query id if it has not been removed yet (since the full admission state only contains running queries)
was:
Ensure consistency between failure detection and registration/Ack of a coordinator by the admission service.
Currently admission service utilizes the statestore membership updates to detect a coordinator going down but it still services RPCs from that coordinator if it is still up and able to contact the admission service.
Using the current mechanisms of statestore updates(IMPALA-10594), admission heartbeats(IMPALA-10590, IMPALA-10720) and coordinator registration(IMPALA-9976) ensure that consistency is maintained between these mechanism.
A possible implementation is:
- Use statestore as the only source of truth.
** Consistency: Only allow a coord to register if it is registered with the statestore
** Atomicity: If the statestore update signals that a coord is down, remove all its state (running and queued queries) before you allow it to register again
> Ensure consistency between failure detection and registration/Ack of a coordinator by the admission service
> -----------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-10866
> URL: https://issues.apache.org/jira/browse/IMPALA-10866
> Project: IMPALA
> Issue Type: Sub-task
> Affects Versions: Impala 4.0.0
> Reporter: Bikramjeet Vig
> Assignee: Bikramjeet Vig
> Priority: Major
>
> Ensure consistency between failure detection and registration/Ack of a coordinator by the admission service.
> Currently admission service utilizes the statestore membership updates to detect a coordinator going down but it still services RPCs from that coordinator if it is still up and able to contact the admission service.
> Using the current mechanisms of statestore updates(IMPALA-10594), admission heartbeats(IMPALA-10590, IMPALA-10720) and coordinator registration(IMPALA-9976) ensure that consistency is maintained between these mechanism.
> A possible implementation is:
> - Use statestore as the only source of truth.
> ** Consistency: Only allow a coord to register if it is registered with the statestore
> ** Atomicity: If the statestore update signals that a coord is down, remove all its state (running and queued queries) before you allow it to register again
> OR
> Eventual consistency: We remove queries between subsequent statestore updates and if the coord comes back up and sends the full admission state, we can update the state of that query id if it has not been removed yet (since the full admission state only contains running queries)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org