You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by st...@apache.org on 2019/12/19 16:20:26 UTC

[impala] branch master updated (094c23b -> 14be8f6)

This is an automated email from the ASF dual-hosted git repository.

stakiar pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.


    from 094c23b  IMPALA-9248: fix test_krpc_rpcz on older kernels
     new 32829b9  IMPALA-9257: Last event id should be advanced if all events are skipped
     new 14be8f6  IMPALA-7117: Lower debug level for HDFS S3 connector back to INFO

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../apache/impala/catalog/events/MetastoreEventsProcessor.java | 10 +++++++---
 fe/src/test/resources/log4j.properties.template                |  3 ---
 tests/custom_cluster/test_event_processing.py                  |  2 ++
 3 files changed, 9 insertions(+), 6 deletions(-)


[impala] 02/02: IMPALA-7117: Lower debug level for HDFS S3 connector back to INFO

Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

stakiar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 14be8f6a30bc715c2b65978e41fc8087d8d08128
Author: Sahil Takiar <ta...@gmail.com>
AuthorDate: Wed Dec 18 12:46:49 2019 -0800

    IMPALA-7117: Lower debug level for HDFS S3 connector back to INFO
    
    This reverts the changes done in https://gerrit.cloudera.org/#/c/10596/
    to set the HDFS S3 connector logs to DEBUG. DEBUG logs drastically increase
    the size of the Impala logs during test runs and are no longer necessary
    given recent stability improvements to the S3A code and the
    Impala-S3Guard integration.
    
    Change-Id: I4b8bf91fbf06e7609e3f92205e71557723163ecf
    Reviewed-on: http://gerrit.cloudera.org:8080/14926
    Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 fe/src/test/resources/log4j.properties.template | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/fe/src/test/resources/log4j.properties.template b/fe/src/test/resources/log4j.properties.template
index 27ba520..31e2a48 100644
--- a/fe/src/test/resources/log4j.properties.template
+++ b/fe/src/test/resources/log4j.properties.template
@@ -26,6 +26,3 @@ log4j.appender.console=org.apache.log4j.ConsoleAppender
 log4j.appender.console.target=System.err
 log4j.appender.console.layout=org.apache.log4j.PatternLayout
 log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
-
-# IMPALA-7008 / IMPALA-7070: Bump S3 log level to get more information.
-log4j.logger.org.apache.hadoop.fs.s3a=DEBUG


[impala] 01/02: IMPALA-9257: Last event id should be advanced if all events are skipped

Posted by st...@apache.org.
This is an automated email from the ASF dual-hosted git repository.

stakiar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 32829b9594a90900a27c8117111a1b3499013e37
Author: Vihang Karajgaonkar <vi...@apache.org>
AuthorDate: Mon Dec 16 15:30:21 2019 -0800

    IMPALA-9257: Last event id should be advanced if all events are skipped
    
    Events processor implements a filtering method which skips certain
    unnecessary events (eg. events on blacklisted dbs and tables).
    However, if the received batch has all the events which are filtered
    out, it fails to update its lastSyncedEventId. This causes unnecessary
    logs being printed in catalog and the same event batch being fetched
    repeatedly.
    
    Testing:
    Modified existing test to compare event id after events on blacklisted
    dbs and tables.
    
    Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32
    Reviewed-on: http://gerrit.cloudera.org:8080/14916
    Reviewed-by: Anurag Mantripragada <an...@cloudera.com>
    Reviewed-by: Quanlong Huang <hu...@gmail.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
 .../apache/impala/catalog/events/MetastoreEventsProcessor.java | 10 +++++++---
 tests/custom_cluster/test_event_processing.py                  |  2 ++
 2 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
index 22c1237..ac50562 100644
--- a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
+++ b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
@@ -584,19 +584,23 @@ public class MetastoreEventsProcessor implements ExternalEventsProcessor {
   /**
    * Process the given list of notification events. Useful for tests which provide a list
    * of events
-   *
-   * @return the last Notification event which was processed.
    */
   @VisibleForTesting
   protected void processEvents(List<NotificationEvent> events)
       throws MetastoreNotificationException {
     NotificationEvent lastProcessedEvent = null;
+    // update the events received metric before returning
+    metrics_.getMeter(EVENTS_RECEIVED_METRIC).mark(events.size());
+    if (events.isEmpty()) return;
     final Timer.Context context =
         metrics_.getTimer(EVENTS_PROCESS_DURATION_METRIC).time();
-    metrics_.getMeter(EVENTS_RECEIVED_METRIC).mark(events.size());
     try {
       List<MetastoreEvent> filteredEvents =
           metastoreEventFactory_.getFilteredEvents(events);
+      if (filteredEvents.isEmpty()) {
+        lastSyncedEventId_.set(events.get(events.size() - 1).getEventId());
+        return;
+      }
       for (MetastoreEvent event : filteredEvents) {
         // synchronizing each event processing reduces the scope of the lock so the a
         // potential reset() during event processing is not blocked for longer than
diff --git a/tests/custom_cluster/test_event_processing.py b/tests/custom_cluster/test_event_processing.py
index 0b7103b..7d67901 100644
--- a/tests/custom_cluster/test_event_processing.py
+++ b/tests/custom_cluster/test_event_processing.py
@@ -52,6 +52,7 @@ class TestEventProcessing(CustomClusterTestSuite):
     event processor does not error out
     """
     try:
+      event_id_before = EventProcessorUtils.get_last_synced_event_id()
       self.run_stmt_in_hive("create database testBlackListedDb")
       self.run_stmt_in_hive("create table testBlackListedDb.testtbl (id int)")
       self.run_stmt_in_hive(
@@ -62,6 +63,7 @@ class TestEventProcessing(CustomClusterTestSuite):
       # wait until all the events generated above are processed
       EventProcessorUtils.wait_for_event_processing(self.hive_client)
       assert EventProcessorUtils.get_event_processor_status() == "ACTIVE"
+      assert EventProcessorUtils.get_last_synced_event_id() > event_id_before
     finally:
       self.run_stmt_in_hive("drop database testBlackListedDb cascade")
       self.run_stmt_in_hive("drop table functional_parquet.testBlackListedTbl")