You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2019/08/02 23:37:00 UTC
[jira] [Commented] (IMPALA-8803) Coordinator should release
admitted memory per-backend rather than per-query
[ https://issues.apache.org/jira/browse/IMPALA-8803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16899281#comment-16899281 ]
Tim Armstrong commented on IMPALA-8803:
---------------------------------------
Yeah, I think there are some queries that fan-in so backends would go idle earlier. Our scheduler isn't really very smart about this now, but could be smarter in future. So I think that's a nice optimisation. I think I agree with your thought process about the batching, I don't think it needs to be totally optimal, but having a log() bound on it is nice.
> Coordinator should release admitted memory per-backend rather than per-query
> ----------------------------------------------------------------------------
>
> Key: IMPALA-8803
> URL: https://issues.apache.org/jira/browse/IMPALA-8803
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> When {{SPOOL_QUERY_RESULTS}} is true, the coordinator backend may be long lived, even though all other backends for the query have completed. Currently, the Coordinator only releases admitted memory when the entire query has completed (include the coordinator fragment) - https://github.com/apache/impala/blob/72c9370856d7436885adbee3e8da7e7d9336df15/be/src/runtime/coordinator.cc#L562
> In order to more aggressively return admitted memory, the coordinator should release memory when each backend for a query completes, rather than waiting for the entire query to complete.
> Releasing memory per backend should be batched because releasing admitted memory in the admission controller requires obtaining a global lock and refreshing the internal stats of the admission controller. Batching will help mitigate any additional overhead from releasing admitted memory per backend.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org