You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "tim geary (JIRA)" <ji...@apache.org> on 2018/06/13 18:17:00 UTC
[jira] [Commented] (IMPALA-5058) Improve concurrency of DDL/DML
operations during catalog updates
[ https://issues.apache.org/jira/browse/IMPALA-5058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16511498#comment-16511498 ]
tim geary commented on IMPALA-5058:
-----------------------------------
Do we know what cdh version this will be in?
> Improve concurrency of DDL/DML operations during catalog updates
> ----------------------------------------------------------------
>
> Key: IMPALA-5058
> URL: https://issues.apache.org/jira/browse/IMPALA-5058
> Project: IMPALA
> Issue Type: Improvement
> Components: Catalog
> Affects Versions: Impala 2.5.0, Impala 2.6.0, Impala 2.7.0
> Reporter: Dimitris Tsirogiannis
> Assignee: Dimitris Tsirogiannis
> Priority: Critical
> Labels: catalog-server, performance, usability
> Fix For: Impala 2.12.0
>
> Attachments: sample-refresh-duration-graph.png
>
>
> Currently, long running DDL/DML operations can block other operations from making progress if they run concurrently with the getCatalogObjects() call that creates catalog updates. The reason is that while getCatalogObjects() holds the lock for its entire duration and also tries to acquire the locks for the tables it processes. If that operation is blocked by another operation on a table then any other, unrelated, catalog write operation cannot make any progress as it cannot acquire the catalog lock which is held by getCatalogObjects().
> From a user's point of view, concurrent DDL/DML operations are executed serially and, consequently, the latency of DDL/DML operations may vary significantly. With the fix for this issue concurrent DDL/DML operations should allow to run concurrently and the throughput of these operations should increase significantly. At the same time, the latency of DDL/DML operations should not depend on any other operations that are running at the same time. It's important to note that when we talk about the latency of an operation it is with respect to the coordinator that initiates the operation; the fix doesn't do anything to improve the latency of broadcasting metadata changes through the statestore. Some common user case where this fix is applicable are the following:
> # Concurrent REFRESH operations on different tables.
> # Concurrent ALTER TABLE operations on different tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org