You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Thomas Tauber-Marshall (Jira)" <ji...@apache.org> on 2019/10/03 19:36:00 UTC
[jira] [Commented] (IMPALA-3160) Queries may not get cancelled if
cancellation pool hits MAX_CANCELLATION_QUEUE_SIZE
[ https://issues.apache.org/jira/browse/IMPALA-3160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16943849#comment-16943849 ]
Thomas Tauber-Marshall commented on IMPALA-3160:
------------------------------------------------
Yes, this is still an issue.
In particular, as the code is currently written, ClusterMembershipMgr will only call 'update_local_server_fn_' a single time for the deleted backend, so if the queries aren't cancelled during the membership update that deletes the backend we will not try to cancel them through that code path again.
However, due to IMPALA-2990, its much less severe than before - previously, such queries would hang indefinitely until manually cancelled. Now, they will eventually time out since the backend that went down will not be reporting status updates.
The default timeout is 10 minutes, even though we know immediately that the backend won't be sending any more updates, so there's definitely room to improve this.
> Queries may not get cancelled if cancellation pool hits MAX_CANCELLATION_QUEUE_SIZE
> -----------------------------------------------------------------------------------
>
> Key: IMPALA-3160
> URL: https://issues.apache.org/jira/browse/IMPALA-3160
> Project: IMPALA
> Issue Type: Bug
> Components: Distributed Exec
> Affects Versions: Impala 2.5.0
> Reporter: Sailesh Mukil
> Assignee: Thomas Tauber-Marshall
> Priority: Minor
> Labels: correctness, downgraded
>
> The ImpalaServer::MembershipCallback() function determines if a backend(s) is down from the topic updates from the statestore. It also cancels all the queries that are already in flight on these failed backends after comparing the failed backend from the topic update to the failed backend in the query_locations_ map which maps backends to queries running on it.
> If the cancellation queue is too large (tracked by MAX_CANCELLATION_QUEUE_SIZE), we do not cancel the queries hoping that by the next heartbeat, the cancellation queue frees up so we can re-try the cancellation of these queries.
> However, by that point we already remove the failed backend from the query_locations_ map. So, the next heartbeat will never find this backend to cancel the queries running on it.
> {code:java}
> // Maps from query id (to be cancelled) to a list of failed Impalads that are
> // the cause of the cancellation.
> map<TUniqueId, vector<TNetworkAddress> > queries_to_cancel; // VVVV: LOCAL MAP
> {
> // Build a list of queries that are running on failed hosts (as evidenced by their
> // absence from the membership list).
> // TODO: crash-restart failures can give false negatives for failed Impala demons.
> lock_guard<mutex> l(query_locations_lock_);
> QueryLocations::const_iterator loc_entry = query_locations_.begin();
> while (loc_entry != query_locations_.end()) {
> if (current_membership.find(loc_entry->first) == current_membership.end()) {
> unordered_set<TUniqueId>::const_iterator query_id = loc_entry->second.begin();
> // Add failed backend locations to all queries that ran on that backend.
> for(; query_id != loc_entry->second.end(); ++query_id) {
> vector<TNetworkAddress>& failed_hosts = queries_to_cancel[*query_id];
> failed_hosts.push_back(loc_entry->first);
> }
> exec_env_->impalad_client_cache()->CloseConnections(loc_entry->first);
> // We can remove the location wholesale once we know backend's failed. To do so
> // safely during iteration, we have to be careful not in invalidate the current
> // iterator, so copy the iterator to do the erase(..) and advance the original.
> QueryLocations::const_iterator failed_backend = loc_entry;
> ++loc_entry;
> // VVVV: WE ERASE THE ENTRY FROM THE GLOBAL MAP HERE.
> query_locations_.erase(failed_backend);
> } else {
> ++loc_entry;
> }
> }
> }
> if (cancellation_thread_pool_->GetQueueSize() + queries_to_cancel.size() >
> MAX_CANCELLATION_QUEUE_SIZE) {
> // Ignore the cancellations - we'll be able to process them on the next heartbeat
> // instead.
> LOG_EVERY_N(WARNING, 60) << "Cancellation queue is full";
> // VVVV: WE DON'T CANCEL HERE AND BY THE NEXT HEARTBEAT, WE WON'T FIND THE FAILED BACKEND AGAIN.
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org