You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by cs...@apache.org on 2019/06/14 15:58:33 UTC

[impala] branch master updated (11b18d7 -> 3a4dce2)

This is an automated email from the ASF dual-hosted git repository.

csringhofer pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.


    from 11b18d7  IMPALA-8654: Log the SQL statement in the Ranger audit log
     new dec14ab  IMPALA-8633 : Insert event should not error when table does not exists
     new 3a4dce2  IMPALA-8436: Do not create materialized_view in Kudu/HBase

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../impala/catalog/CatalogServiceCatalog.java      |   2 +-
 .../impala/catalog/events/MetastoreEvents.java     |  32 +--
 .../apache/impala/service/CatalogOpExecutor.java   |  45 +----
 .../java/org/apache/impala/util/MetaStoreUtil.java |  44 ++++-
 .../events/MetastoreEventsProcessorTest.java       | 218 +++++++++++----------
 .../datasets/functional/schema_constraints.csv     |   6 +
 6 files changed, 193 insertions(+), 154 deletions(-)


[impala] 01/02: IMPALA-8633 : Insert event should not error when table does not exists

Posted by cs...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

csringhofer pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit dec14abbd46467a73d1159508971e04da68fa508
Author: Vihang Karajgaonkar <vi...@cloudera.com>
AuthorDate: Thu Jun 6 11:40:57 2019 -0700

    IMPALA-8633 : Insert event should not error when table does not exists
    
    When a insert event is received by the events processor it is possible
    that the table has already been removed from metastore. In such cases,
    the reloadTable method throws a NoSuchObjectException which is
    unhandled. This causes EventProcessor to go in NEEDS_INVALIDATE state.
    The patch adds exception handling for such cases so that the error can
    be ignored and a warning is logged.
    
    Ideally, if the table has been removed from metastore, this insert event
    should be followed by a drop-table event and hence the table is cleaned
    up from the catalog subsequently. Hence Event processor does not need to
    remove the table from the catalog.
    
    Also, the patch adds the exception trace to the error messages during
    processing of Insert events to improve debugging ability in case of
    such errors.
    
    Testing:
    
    Refactored the existing test for insert events into util methods
    which can be reused for other tests. Added a new test case which
    generates a insert event on a table which has been dropped. This test
    reproduces the scenario seen by debugging test failures in IMPALA-8567.
    The newly added test succeeds after the patch.
    
    Change-Id: I961cd7cbede4c248dba538c7fabb4bc708e49693
    Reviewed-on: http://gerrit.cloudera.org:8080/13548
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 .../impala/catalog/CatalogServiceCatalog.java      |   2 +-
 .../impala/catalog/events/MetastoreEvents.java     |  32 +--
 .../apache/impala/service/CatalogOpExecutor.java   |  45 +----
 .../java/org/apache/impala/util/MetaStoreUtil.java |  44 ++++-
 .../events/MetastoreEventsProcessorTest.java       | 218 +++++++++++----------
 5 files changed, 187 insertions(+), 154 deletions(-)

diff --git a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
index 39a255d..8c4678f 100644
--- a/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
+++ b/fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
@@ -2158,7 +2158,7 @@ public class CatalogServiceCatalog extends Catalog {
    * otherwise. Throws CatalogException if reloadTable() is unsuccessful. Throws
    * DatabaseNotFoundException if Db doesn't exist.
    */
-  public boolean refreshTableIfExists(String dbName, String tblName)
+  public boolean reloadTableIfExists(String dbName, String tblName)
       throws CatalogException {
     Table table = getTable(dbName, tblName);
     if (table == null || table instanceof IncompleteTable) return false;
diff --git a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
index 7a6457d..07251c9 100644
--- a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
+++ b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
@@ -31,6 +31,7 @@ import java.util.Map;
 import org.apache.hadoop.hive.metastore.api.Database;
 import org.apache.hadoop.hive.metastore.api.NotificationEvent;
 import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
 import org.apache.hadoop.hive.metastore.messaging.AddPartitionMessage;
 import org.apache.hadoop.hive.metastore.messaging.AlterPartitionMessage;
 import org.apache.hadoop.hive.metastore.messaging.CreateTableMessage;
@@ -48,6 +49,7 @@ import org.apache.impala.catalog.DatabaseNotFoundException;
 import org.apache.impala.catalog.Db;
 import org.apache.impala.catalog.Table;
 import org.apache.impala.catalog.TableNotFoundException;
+import org.apache.impala.catalog.TableLoadingException;
 import org.apache.impala.common.Metrics;
 import org.apache.impala.common.Pair;
 import org.apache.impala.common.Reference;
@@ -775,7 +777,8 @@ public class MetastoreEvents {
           insertPartition_);
       try {
         // Ignore event if table or database is not in catalog. Throw exception if
-        // refresh fails.
+        // refresh fails. If the partition does not exist in metastore the reload
+        // method below removes it from the catalog
         if (!catalog_.reloadPartitionIfExists(dbName_, tblName_, tPartSpec)) {
           debugLog("Refresh of table {} partition {} after insert "
                   + "event failed as the table is not present in the catalog.",
@@ -795,7 +798,7 @@ public class MetastoreEvents {
                 + "partition on table {} partition {} failed. Event processing cannot "
                 + "continue. Issue and invalidate command to reset the event processor "
                 + "state.", getFullyQualifiedTblName(),
-            constructPartitionStringFromTPartitionSpec(tPartSpec)));
+            constructPartitionStringFromTPartitionSpec(tPartSpec)), e);
       }
     }
 
@@ -808,9 +811,9 @@ public class MetastoreEvents {
       try {
         // Ignore event if table or database is not in the catalog. Throw exception if
         // refresh fails.
-        if (!catalog_.refreshTableIfExists(dbName_, tblName_)) {
+        if (!catalog_.reloadTableIfExists(dbName_, tblName_)) {
           debugLog("Automatic refresh table {} failed as the table is not "
-              + "present in the catalog. ", getFullyQualifiedTblName());
+              + "present either catalog or metastore.", getFullyQualifiedTblName());
         } else {
           infoLog("Table {} has been refreshed after insert.",
               getFullyQualifiedTblName());
@@ -819,10 +822,17 @@ public class MetastoreEvents {
         debugLog("Automatic refresh of table {} insert failed as the "
             + "database is not present in the catalog.", getFullyQualifiedTblName());
       } catch (CatalogException e) {
-        throw new MetastoreNotificationNeedsInvalidateException(
-            debugString("Refresh table {} failed. Event processing "
-                + "cannot continue. Issue an invalidate metadata command to reset "
-                + "the event processor state.", getFullyQualifiedTblName()));
+        if (e instanceof TableLoadingException &&
+            e.getCause() instanceof NoSuchObjectException) {
+          LOG.warn(
+              "Ignoring the refresh of the table since the table does"
+                  + " not exist in metastore anymore");
+        } else {
+          throw new MetastoreNotificationNeedsInvalidateException(
+              debugString("Refresh table {} failed. Event processing "
+                  + "cannot continue. Issue an invalidate metadata command to reset "
+                  + "the event processor state.", getFullyQualifiedTblName()), e);
+        }
       }
     }
   }
@@ -1375,7 +1385,7 @@ public class MetastoreEvents {
         throw new MetastoreNotificationNeedsInvalidateException(debugString("Failed to "
                 + "refresh newly added partitions of table {}. Event processing cannot "
                 + "continue. Issue an invalidate command to reset event processor.",
-            getFullyQualifiedTblName()));
+            getFullyQualifiedTblName()), e);
       }
     }
 
@@ -1461,7 +1471,7 @@ public class MetastoreEvents {
                 + "partition on table {} partition {} failed. Event processing cannot "
                 + "continue. Issue and invalidate command to reset the event processor "
                 + "state.", getFullyQualifiedTblName(),
-            constructPartitionStringFromTPartitionSpec(tPartSpec)));
+            constructPartitionStringFromTPartitionSpec(tPartSpec)), e);
       }
     }
 
@@ -1547,7 +1557,7 @@ public class MetastoreEvents {
         throw new MetastoreNotificationNeedsInvalidateException(debugString("Failed to "
             + "drop some partitions from table {} after a drop partitions event. Event "
                 + "processing cannot continue. Issue an invalidate command to reset "
-            + "event processor state.", getFullyQualifiedTblName()));
+            + "event processor state.", getFullyQualifiedTblName()), e);
       }
     }
 
diff --git a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
index 63d4166..2dfca63 100644
--- a/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
+++ b/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
@@ -45,9 +45,6 @@ import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
 import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
 import org.apache.hadoop.hive.metastore.api.Database;
 import org.apache.hadoop.hive.metastore.api.FieldSchema;
-import org.apache.hadoop.hive.metastore.api.FireEventRequest;
-import org.apache.hadoop.hive.metastore.api.FireEventRequestData;
-import org.apache.hadoop.hive.metastore.api.InsertEventRequestData;
 import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
 import org.apache.hadoop.hive.metastore.api.Partition;
 import org.apache.hadoop.hive.metastore.api.PrincipalType;
@@ -3848,7 +3845,14 @@ public class CatalogOpExecutor {
             filesPostInsert.size(), table.getTableName(), part.getPartitionName());
       }
       if (deltaFiles != null || isInsertOverwrite) {
-        fireInsertEvent(table, partVals, deltaFiles, isInsertOverwrite);
+        try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
+          MetaStoreUtil
+              .fireInsertEvent(metaStoreClient.getHiveClient(), table.getDb().getName(),
+                  table.getName(), partVals, deltaFiles, isInsertOverwrite);
+        } catch (Exception e) {
+          LOG.error("Failed to fire insert event. Some tables might not be"
+              + " refreshed on other impala clusters.", e);
+        }
       }
       else {
         LOG.info("No new files were created, and is not a replace. Skipping "
@@ -3858,39 +3862,6 @@ public class CatalogOpExecutor {
   }
 
   /**
-   *  Fires an insert event to HMS notification log. For partitioned table, each
-   *  existing partition touched by the insert will fire a separate insert event.
-   *
-   * @param newFiles Set of all the 'new' files added by this insert. This is empty in
-   * case of insert overwrite.
-   * @param partVals List of partition values corresponding to the partition keys in
-   * a partitioned table. This is null for non-partitioned table.
-   * @param isOverwrite If true, sets the 'replace' flag to true indicating that the
-   * operation was an insert overwrite in the notification log. Will set the same to
-   * false otherwise.
-   */
-  private void fireInsertEvent(Table tbl, List<String> partVals,
-      Set<String> newFiles, boolean isOverwrite) {
-    LOG.debug("Firing an insert event for {}.", tbl.getName());
-    FireEventRequestData data = new FireEventRequestData();
-    InsertEventRequestData insertData = new InsertEventRequestData();
-    data.setInsertData(insertData);
-    FireEventRequest rqst = new FireEventRequest(true, data);
-    rqst.setDbName(tbl.getDb().getName());
-    rqst.setTableName(tbl.getName());
-    insertData.setFilesAdded(new ArrayList<>(newFiles));
-    insertData.setReplace(isOverwrite);
-    rqst.setPartitionVals(partVals);
-
-    try (MetaStoreClient msClient = catalog_.getMetaStoreClient()) {
-      msClient.getHiveClient().fireListenerEvent(rqst);
-    } catch (Exception e) {
-      LOG.error("Failed to fire insert event. Some tables might not be"
-          + " refreshed on other impala clusters.", e);
-    }
-  }
-
-  /**
    * Returns an existing, loaded table from the Catalog. Throws an exception if any
    * of the following are true:
    * - The table does not exist
diff --git a/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java b/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
index ee026d8..a1e58c0 100644
--- a/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
+++ b/fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java
@@ -17,16 +17,21 @@
 
 package org.apache.impala.util;
 
+import java.util.ArrayList;
 import java.util.HashSet;
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
 
+import java.util.Collection;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.metastore.IMetaStoreClient;
 import org.apache.hadoop.hive.metastore.Warehouse;
 import org.apache.hadoop.hive.metastore.api.ConfigValSecurityException;
 import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.FireEventRequest;
+import org.apache.hadoop.hive.metastore.api.FireEventRequestData;
+import org.apache.hadoop.hive.metastore.api.InsertEventRequestData;
 import org.apache.hadoop.hive.metastore.api.MetaException;
 import org.apache.hadoop.hive.metastore.api.Partition;
 import org.apache.hadoop.hive.metastore.api.Table;
@@ -34,7 +39,6 @@ import org.apache.impala.catalog.CatalogException;
 import org.apache.impala.catalog.HdfsTable;
 import org.apache.impala.common.AnalysisException;
 import org.apache.impala.compat.MetastoreShim;
-import org.apache.log4j.Logger;
 import org.apache.thrift.TException;
 
 import org.apache.impala.thrift.TColumn;
@@ -43,12 +47,14 @@ import com.google.common.base.Preconditions;
 import com.google.common.base.Splitter;
 import com.google.common.collect.Lists;
 import com.google.common.collect.Sets;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * Utility methods for interacting with the Hive Metastore.
  */
 public class MetaStoreUtil {
-  private static final Logger LOG = Logger.getLogger(MetaStoreUtil.class);
+  private static final Logger LOG = LoggerFactory.getLogger(MetaStoreUtil.class);
 
   // Maximum comment length, e.g., for columns, that can be stored in the HMS.
   // This number is a lower bound of the constraint set in the HMS DB schema,
@@ -306,4 +312,38 @@ public class MetaStoreUtil {
     }
     return partVals;
   }
+
+  /**
+   *  Fires an insert event to HMS notification log. For partitioned table, each
+   *  existing partition touched by the insert will fire a separate insert event.
+   *
+   * @param msClient Metastore client
+   * @param newFiles Set of all the 'new' files added by this insert. This is empty in
+   * case of insert overwrite.
+   * @param partVals List of partition values corresponding to the partition keys in
+   * a partitioned table. This is null for non-partitioned table.
+   * @param isOverwrite If true, sets the 'replace' flag to true indicating that the
+   * operation was an insert overwrite in the notification log. Will set the same to
+   * false otherwise.
+   */
+  public static void fireInsertEvent(IMetaStoreClient msClient,
+      String dbName, String tblName, List<String> partVals,
+      Collection<String> newFiles, boolean isOverwrite) throws TException {
+    Preconditions.checkNotNull(msClient);
+    Preconditions.checkNotNull(dbName);
+    Preconditions.checkNotNull(tblName);
+    Preconditions.checkNotNull(newFiles);
+    LOG.debug("Firing an insert event for {}", tblName);
+    FireEventRequestData data = new FireEventRequestData();
+    InsertEventRequestData insertData = new InsertEventRequestData();
+    data.setInsertData(insertData);
+    FireEventRequest rqst = new FireEventRequest(true, data);
+    rqst.setDbName(dbName);
+    rqst.setTableName(tblName);
+    insertData.setFilesAdded(new ArrayList<>(newFiles));
+    insertData.setReplace(isOverwrite);
+    if (partVals != null) rqst.setPartitionVals(partVals);
+
+    msClient.fireListenerEvent(rqst);
+  }
 }
diff --git a/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java b/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
index eb426b3..d9bce89 100644
--- a/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
+++ b/fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
@@ -28,7 +28,6 @@ import static org.junit.Assert.assertNotNull;
 import static org.junit.Assert.assertNull;
 import static org.junit.Assert.assertTrue;
 import static org.junit.Assert.fail;
-import static org.mockito.Mockito.when;
 
 import com.google.common.base.Preconditions;
 import com.google.common.collect.Iterables;
@@ -45,10 +44,10 @@ import java.util.Set;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.atomic.AtomicLong;
 
+import org.apache.commons.lang.RandomStringUtils;
 import org.apache.hadoop.fs.FSDataOutputStream;
 import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.hive.metastore.IMetaStoreClient;
 import org.apache.hadoop.hive.metastore.api.CurrentNotificationEventId;
 import org.apache.hadoop.hive.metastore.api.Database;
 import org.apache.hadoop.hive.metastore.api.FieldSchema;
@@ -75,7 +74,6 @@ import org.apache.impala.catalog.MetaStoreClientPool.MetaStoreClient;
 import org.apache.impala.catalog.Table;
 import org.apache.impala.catalog.events.ConfigValidator.ValidationResult;
 import org.apache.impala.catalog.events.MetastoreEvents.AlterTableEvent;
-import org.apache.impala.catalog.events.MetastoreEvents.InsertEvent;
 import org.apache.impala.catalog.events.MetastoreEvents.MetastoreEvent;
 import org.apache.impala.catalog.events.MetastoreEvents.MetastoreEventPropertyKey;
 import org.apache.impala.catalog.events.MetastoreEvents.MetastoreEventType;
@@ -136,7 +134,6 @@ import org.junit.Test;
 
 import com.google.common.collect.Lists;
 
-import org.mockito.Mockito;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -605,7 +602,7 @@ public class MetastoreEventsProcessorTest {
    * calls insertEvent tests on them.
    */
   @Test
-  public void testInsertEvents() throws TException, ImpalaException {
+  public void testInsertEvents() throws TException, ImpalaException, IOException {
     // Test insert into partition
     createDatabase(TEST_DB_NAME, null);
     String tableToInsertPart = "tbl_to_insert_part";
@@ -619,13 +616,62 @@ public class MetastoreEventsProcessorTest {
   }
 
   /**
+   * Test generates a sequence of create_table, insert and drop_table in the event stream
+   * to make sure when the insert event is processed on a removed table, it doesn't cause
+   * any issues with the event processing.
+   */
+  @Test
+  public void testInsertEventOnRemovedTable()
+      throws Exception {
+    createDatabaseFromImpala(TEST_DB_NAME, "");
+    final String createInsertDropTable = "tbl_create_insert_drop";
+    createTableFromImpala(TEST_DB_NAME, createInsertDropTable, null, false);
+    org.apache.hadoop.hive.metastore.api.Table msTbl;
+    try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
+      msTbl =
+          metaStoreClient.getHiveClient().getTable(TEST_DB_NAME, createInsertDropTable);
+    }
+    simulateInsertIntoTableFromFS(msTbl, 2, null, false);
+    dropTable(createInsertDropTable);
+    // this will generate create db, create_table, insert and droptable events
+    assertEquals(4, eventsProcessor_.getNextMetastoreEvents().size());
+    eventsProcessor_.processEvents();
+    // make sure that the insert event on a dropped table does not cause a error during
+    // event processing
+    assertEquals(EventProcessorStatus.ACTIVE, eventsProcessor_.getStatus());
+  }
+
+  /**
+   * Util method to create empty files in a given path
+   * @param parentPath
+   * @param fileNamePrefix
+   * @param totalNumberOfFilesToAdd
+   * @return
+   * @throws IOException
+   */
+  private List<String> addFilesToDirectory(Path parentPath, String fileNamePrefix,
+      int totalNumberOfFilesToAdd, boolean isOverwrite) throws IOException {
+    List<String> newFiles = new ArrayList<>();
+    final FileSystem fs = parentPath.getFileSystem(FileSystemUtil.getConfiguration());
+    if (isOverwrite && fs.exists(parentPath)) fs.delete(parentPath, true);
+    for (int i = 0; i < totalNumberOfFilesToAdd; i++) {
+      Path filename = new Path(parentPath,
+          fileNamePrefix + RandomStringUtils.random(5, true, true).toUpperCase());
+      try (FSDataOutputStream out = fs.create(filename)) {
+        newFiles.add(filename.getName());
+      }
+    }
+    return newFiles;
+  }
+
+  /**
    * Helper to test insert events. Creates a fake InsertEvent notification in the
    * catalog and processes it. To simulate an insert, we load a file using FS APIs and
    * verify the new file shows up after table/partition refresh.
    */
   public void testInsertEvents(String dbName, String tblName,
       boolean isPartitionInsert) throws TException,
-      ImpalaException {
+      ImpalaException, IOException {
 
     if (isPartitionInsert) {
       // Add a partition
@@ -635,106 +681,72 @@ public class MetastoreEventsProcessorTest {
     }
     eventsProcessor_.processEvents();
 
-    // To simulate an insert, load a file into the partition location.
-    String parentPathString =
-        "/test-warehouse/" + dbName +".db/" + tblName;
-    String filePathString = isPartitionInsert ? "/p1=testPartVal/testFile.0" :
-        "/testFile.0";
-    Path parentPath =
-        FileSystemUtil.createFullyQualifiedPath(new Path(parentPathString));
-    FileSystem fs = null;
-    FSDataOutputStream out = null;
-    List<String> newFiles = new ArrayList<>(1);
-    try {
-      fs = parentPath.getFileSystem(FileSystemUtil.getConfiguration());
-      // put a test file into the location.
-      out = fs.create(new Path(parentPathString + filePathString));
-      newFiles.add(filePathString);
-      org.apache.hadoop.hive.metastore.api.Partition partition = null ;
-      if (isPartitionInsert) {
-        // Get the partition from metastore. This should now contain the new file.
-        try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
-          partition = metaStoreClient.getHiveClient().getPartition(dbName, tblName, "p1"
-              + "=testPartVal");
-        }
-      }
-      // Simulate a load table
-      Table tbl = catalog_.getOrLoadTable(dbName, tblName);
-      NotificationEvent fakeInsertEvent = createFakeInsertEvent(tbl.getMetaStoreTable(),
-          partition, false, newFiles);
-      InsertEvent insertEvent = new InsertEvent(catalog_,
-          eventsProcessor_.getMetrics(), fakeInsertEvent);
-
-      // Process this event, this should refresh the table.
-      insertEvent.process();
-
-      // Now check if the table is refreshed by checking the files size. A partition
-      // refresh will make the new file show up in the partition. NOTE: This is same
-      // for table and partition inserts as impala treats a non-partitioned table as a
-      // table with a single partition.
-      Table tblAfterInsert = catalog_.getTable(dbName, tblName);
-      Collection<? extends FeFsPartition> partsAfterInsert =
-          FeCatalogUtils.loadAllPartitions((HdfsTable) tblAfterInsert);
-      assertTrue("Partition not found after insert.",
-          partsAfterInsert.size() > 0);
-      FeFsPartition singlePart =
-          Iterables.getOnlyElement((List<FeFsPartition>) partsAfterInsert);
-      Set<String> filesAfterInsertForTable =
-          (((HdfsPartition) singlePart).getFileNames());
-      assertTrue("File count mismatch after insert.",
-          filesAfterInsertForTable.size() == 1);
-
-      // Create another event for overwrite
-      NotificationEvent fakeInsertOverwriteEvent =
-          createFakeInsertEvent(tbl.getMetaStoreTable(),
-              partition, true, newFiles);
-      InsertEvent insertOverwriteEvent = new InsertEvent(catalog_,
-          eventsProcessor_.getMetrics(), fakeInsertOverwriteEvent);
-      insertOverwriteEvent.process();
-
-      // Overwrite is expected to behave similarly.
-      Table tblAfterInsertOverwrite = catalog_.getTable(dbName, tblName);
-      Collection<? extends FeFsPartition> partsAfterInsertOverwrite =
-          FeCatalogUtils.loadAllPartitions((HdfsTable) tblAfterInsertOverwrite);
-      assertTrue("Partition not found after insert.",
-          partsAfterInsertOverwrite.size() > 0);
-      FeFsPartition singlePartAfterInsertOverwrite =
-          Iterables.getOnlyElement((List<FeFsPartition>) partsAfterInsertOverwrite);
-      Set<String> filesAfterInsertOverwriteForTable =
-          (((HdfsPartition) singlePartAfterInsertOverwrite).getFileNames());
-      assertTrue("File count mismatch after insert.",
-          filesAfterInsertOverwriteForTable.size() == 1);
-    } catch (IOException e) {
-      throw new MetastoreNotificationException(e);
-    } finally {
-      if (out != null) {
-        try {
-          out.close();
-        } catch (IOException e) {
-          throw new MetastoreNotificationException(e);
-        } finally {
-          FileSystemUtil.deleteIfExists(parentPath);
-        }
+    // Simulate a load table
+    Table tbl = catalog_.getOrLoadTable(dbName, tblName);
+    Partition partition = null;
+    if (isPartitionInsert) {
+      // Get the partition from metastore. This should now contain the new file.
+      try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
+        partition = metaStoreClient.getHiveClient().getPartition(dbName, tblName, "p1"
+            + "=testPartVal");
       }
     }
-  }
-
-  /**
-   * Helper to create a fake insert notification event.
-   */
-  private NotificationEvent createFakeInsertEvent(
+    // first insert 3 files
+    assertFalse("Table must be already loaded to verify correctness",
+        tbl instanceof IncompleteTable);
+    simulateInsertIntoTableFromFS(tbl.getMetaStoreTable(), 3,
+        partition, true);
+    verifyNumberOfFiles(tbl, 3);
+
+    // another insert which adds 2 more files; total number of files will now be 5
+    simulateInsertIntoTableFromFS(tbl.getMetaStoreTable(), 2,
+        partition, false);
+    verifyNumberOfFiles(tbl, 5);
+
+    // create a insert overwrite op now to reset the file count to 1
+    simulateInsertIntoTableFromFS(tbl.getMetaStoreTable(), 1,
+        partition, true);
+    verifyNumberOfFiles(tbl, 1);
+
+    // create insert overwrite again to add 2 files
+    simulateInsertIntoTableFromFS(tbl.getMetaStoreTable(), 2,
+        partition, true);
+    verifyNumberOfFiles(tbl, 2);
+  }
+
+  private void verifyNumberOfFiles(Table tbl, int expectedNumberOfFiles)
+      throws DatabaseNotFoundException {
+    // Now check if the table is refreshed by checking the files size. A partition
+    // refresh will make the new file show up in the partition. NOTE: This is same
+    // for table and partition inserts as impala treats a non-partitioned table as a
+    // table with a single partition.
+    eventsProcessor_.processEvents();
+    Table tblAfterInsert = catalog_.getTable(tbl.getDb().getName(), tbl.getName());
+    assertFalse(tblAfterInsert instanceof IncompleteTable);
+    Collection<? extends FeFsPartition> partsAfterInsert =
+        FeCatalogUtils.loadAllPartitions((HdfsTable) tblAfterInsert);
+    assertTrue("Partition not found after insert.",
+        partsAfterInsert.size() > 0);
+    FeFsPartition singlePart =
+        Iterables.getOnlyElement((List<FeFsPartition>) partsAfterInsert);
+    Set<String> filesAfterInsertForTable =
+        (((HdfsPartition) singlePart).getFileNames());
+    assertEquals("File count mismatch after insert.",
+        expectedNumberOfFiles, filesAfterInsertForTable.size());
+  }
+
+  private void simulateInsertIntoTableFromFS(
       org.apache.hadoop.hive.metastore.api.Table msTbl,
-      org.apache.hadoop.hive.metastore.api.Partition partition,
-      boolean isInsertOverwrite, List<String> newFiles) {
-    NotificationEvent fakeEvent = new NotificationEvent();
-    fakeEvent.setTableName(msTbl.getTableName());
-    fakeEvent.setDbName(msTbl.getDbName());
-    fakeEvent.setEventId(eventIdGenerator.incrementAndGet());
-    fakeEvent.setMessage(MetastoreShim.buildInsertMessage(msTbl, partition,
-       isInsertOverwrite, newFiles).toString());
-    fakeEvent.setEventType("INSERT");
-    return fakeEvent;
-
+      int totalNumberOfFilesToAdd, Partition partition,
+      boolean isOverwrite) throws IOException, TException {
+    Path parentPath = partition == null ? new Path(msTbl.getSd().getLocation())
+        : new Path(partition.getSd().getLocation());
+    List <String> newFiles = addFilesToDirectory(parentPath, "testFile.",
+        totalNumberOfFilesToAdd, isOverwrite);
+    try (MetaStoreClient metaStoreClient = catalog_.getMetaStoreClient()) {
+      MetaStoreUtil.fireInsertEvent(metaStoreClient.getHiveClient(), msTbl.getDbName(),
+          msTbl.getTableName(), null, newFiles, isOverwrite);
+    }
   }
 
   /**


[impala] 02/02: IMPALA-8436: Do not create materialized_view in Kudu/HBase

Posted by cs...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

csringhofer pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 3a4dce24a9b0113b049b2f22e4b34d5ac8153416
Author: Csaba Ringhofer <cs...@cloudera.com>
AuthorDate: Thu Jun 13 18:05:26 2019 +0200

    IMPALA-8436: Do not create materialized_view in Kudu/HBase
    
    IMPALA-8436 broke the dataload on Hive 3, because it tried to
    create materialized_view for file formats where its base table
    does not exist.
    
    Change-Id: Iefe2b32f4cafe9988051b69cefcc1c1f5d4c79e7
    Reviewed-on: http://gerrit.cloudera.org:8080/13634
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 testdata/datasets/functional/schema_constraints.csv | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/testdata/datasets/functional/schema_constraints.csv b/testdata/datasets/functional/schema_constraints.csv
index d66e60d..0546c73 100644
--- a/testdata/datasets/functional/schema_constraints.csv
+++ b/testdata/datasets/functional/schema_constraints.csv
@@ -248,3 +248,9 @@ table_name:full_transactional_table, constraint:restrict_to, table_format:orc/de
 # Insert-only transactional tables only work for file-format based tables
 table_name:insert_only_transactional_table, constraint:exclude, table_format:hbase/none/none
 table_name:insert_only_transactional_table, constraint:exclude, table_format:kudu/none/none
+
+# 'materialized_view' is based on 'insert_only_transactional_table' from the same
+# database, so it needs to be excluded where 'insert_only_transactional_table' is
+# excluded.
+table_name:materialized_view, constraint:exclude, table_format:hbase/none/none
+table_name:materialized_view, constraint:exclude, table_format:kudu/none/none