[jira] [Commented] (IMPALA-4137) Rolling restart of Impala

["Tim Armstrong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-4137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640246#comment-16640246 ] 

Tim Armstrong commented on IMPALA-4137:
---------------------------------------

Just an update here. We have the graceful shutdown command, which is an important building block. Alan's comments about statestore and catalogd require some addressing (since we don't tolerate restarts of those well in general):

Statestore:
I think statestore restarts should be tolerated OK in theory, but there's a bug in practice: IMPALA-7665. The other caveat is if the statestore is down a long time and another impalad goes down, we don't detect it and remove it from the cluster membership (not sure if we have a JIRA for that yet). So if we fix IMPALA-7665 we should be able to restart the statestore, so long as we're not bouncing impala daemons at the same time. So we're not that far off from clean statestore restarts, but statestore HA where we can tolerate extended outages is a bigger task.

Catalog:
Also in theory we should be able to restart the catalog without disrupting running queries and only delaying queries that need new metadata. This would need more testing to make sure that it works well in all cases. I think the metadata delay is also somewhat unacceptable for true rolling restart. The likely solution to that is IMPALA-7127 removing the catalog dependency and having coordinators load data directly.

> Rolling restart of Impala
> -------------------------
>
>                 Key: IMPALA-4137
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4137
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Distributed Exec
>    Affects Versions: Impala 2.2.4
>            Reporter: Alan Jackoway
>            Priority: Major
>              Labels: resource-management
>
> Apologies if a jira exists for this. I could not find one.
> It would be very helpful to us to be able to rolling restart (and hopefully rolling upgrade) Impala.
> Based on my understanding of impala internals, I think this would require:
> * Highly available statestore and catalog. Currently catalog's metadata reload is the long pole in our impala restarts.
> * Impalads being able to stop without killing queries they are working on. Most of our queries are short so for us it would be sufficient to give the impala daemon a way to stop taking new work, then restart when it completes all work it has assigned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org