You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Dimitris Tsirogiannis (JIRA)" <ji...@apache.org> on 2017/09/26 20:37:00 UTC
[jira] [Resolved] (IMPALA-5538) Use explicit catalog versions for
deleted objects in catalog updates
[ https://issues.apache.org/jira/browse/IMPALA-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dimitris Tsirogiannis resolved IMPALA-5538.
-------------------------------------------
Resolution: Fixed
Fix Version/s: Impala 2.11.0
Reviewed-on: http://gerrit.cloudera.org:8080/7731
Reviewed-by: Dimitris Tsirogiannis <dt...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog-util.cc
M be/src/catalog/catalog-util.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/admission-controller.h
M be/src/scheduling/scheduler-test-util.cc
M be/src/scheduling/scheduler.cc
M be/src/service/impala-server.cc
M be/src/statestore/statestore.cc
M be/src/statestore/statestore.h
M common/thrift/CatalogInternalService.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/org/apache/impala/catalog/CatalogDeltaLog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M tests/statestore/test_statestore.py
21 files changed, 480 insertions(+), 441 deletions(-)
> Use explicit catalog versions for deleted objects in catalog updates
> --------------------------------------------------------------------
>
> Key: IMPALA-5538
> URL: https://issues.apache.org/jira/browse/IMPALA-5538
> Project: IMPALA
> Issue Type: Sub-task
> Components: Catalog
> Affects Versions: Impala 2.9.0
> Reporter: Dimitris Tsirogiannis
> Assignee: Dimitris Tsirogiannis
> Priority: Critical
> Labels: concurrency
> Fix For: Impala 2.11.0
>
>
> Catalog update topic updates include both additions and deletions. Even though added objects have a catalog version, deleted objects use the max catalog object version from the update message and not the catalog version that was assigned to the associated object when it was deleted. That works fine assuming the catalog updates do not have gaps (all updates between versions t1 and t2 are included). With the fix for IMPALA-5058, the semantics of what's included in the catalog updates will change, resulting in metadata inconsistencies if we don't assign the correct versions to deleted objects.
> Disclaimer: The above mentioned "versions" indicate the catalog versions set by the catalog server and not the statestore topic versions.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)