[impala] 02/02: IMPALA-10080: Skip loading HDFS cache pools for non-HDFS file systems

[st...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

stigahuang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 568b3394b2945d684d8fdb6c4f4e1f33cbf98898
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Thu Aug 13 10:57:21 2020 +0800

    IMPALA-10080: Skip loading HDFS cache pools for non-HDFS file systems
    
    In global invalidate metadata, we always load HDFS cache pools using the
    CachePoolReader. Actually, it only works for HDFS file systems, not for
    other systems like S3 or local, etc. We already handle this in
    CatalogServiceCatalog#CatalogServiceCatalog(). This patch adds a check
    in CatalogServiceCatalog#reset() to skip loading cache pools if it's not
    a true HDFS file system.
    
    Tests
    - Ran tests on S3. Verified that the IllegalStateException doesn't
      exists anymore.
    
    Change-Id: Ib243d349177e1b982b313dd6e87ecc2ef4dfc3d8
    Reviewed-on: http://gerrit.cloudera.org:8080/16335
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 .../org/apache/impala/catalog/CatalogServiceCatalog.java     | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
index f6cfeb0..e51f56f 100644
--- a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
@@ -1690,8 +1690,16 @@ public class CatalogServiceCatalog extends Catalog {
     // pause the event processing since the cache is anyways being cleared
     metastoreEventProcessor_.pause();
     // Update the HDFS cache pools
-    CachePoolReader reader = new CachePoolReader(true);
-    reader.run();
+    try {
+      // We want only 'true' HDFS filesystems to poll the HDFS cache (i.e not S3,
+      // local, etc.)
+      if (FileSystemUtil.getDefaultFileSystem() instanceof DistributedFileSystem) {
+        CachePoolReader reader = new CachePoolReader(true);
+        reader.run();
+      }
+    } catch (IOException e) {
+      LOG.error("Couldn't identify the default FS. Cache Pool reader will be disabled.");
+    }
     versionLock_.writeLock().lock();
     // In case of an empty new catalog, the version should still change to reflect the
     // reset operation itself and to unblock impalads by making the catalog version >