[Impala-ASF-CR] IMPALA-9045: Filter base directories of open/aborted compactions

["Zoltan Borok-Nagy (Code Review)" <ge...@cloudera.org>]

Hello Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/14547

to look at the new patch set (#2).

Change subject: IMPALA-9045: Filter base directories of open/aborted compactions
......................................................................

IMPALA-9045: Filter base directories of open/aborted compactions

Base directories are in the format of base_<write_id>_<transaction_id>.
The <transaction_id> part helps to decide whether a base directory
is fully written, or is it still being written by a compaction job.
Compaction jobs don't increase the write id of a table, hence the
<write_id> part cannot be used for that.

Before this commit Impala didn't check the validity of <transaction_id>,
therefore it might read the contents of a half-written base directory.
With this change Impala retrieves the valid transaction list from HMS
and checks if <transaction_id> is committed.

Testing
 * Added an e2e test that simulates in-progress compactions

Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6
---
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java
M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java
M tests/query_test/test_acid.py
12 files changed, 178 insertions(+), 43 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/14547/2
-- 
To view, visit http://gerrit.cloudera.org:8080/14547
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Idb895df38bc075e4767e44a6887dbe3000a19ea6
Gerrit-Change-Number: 14547
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy <bo...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Zoltan Borok-Nagy <bo...@cloudera.com>