This is an automated email from the ASF dual-hosted git repository.
stakiar pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git.
from 094c23b IMPALA-9248: fix test_krpc_rpcz on older kernels
new 32829b9 IMPALA-9257: Last event id should be advanced if all events are skipped
new 14be8f6 IMPALA-7117: Lower debug level for HDFS S3 connector back to INFO
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../apache/impala/catalog/events/MetastoreEventsProcessor.java | 10 +++++++---
fe/src/test/resources/log4j.properties.template | 3 ---
tests/custom_cluster/test_event_processing.py | 2 ++
3 files changed, 9 insertions(+), 6 deletions(-)
This is an automated email from the ASF dual-hosted git repository.
stakiar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 14be8f6a30bc715c2b65978e41fc8087d8d08128
Author: Sahil Takiar <ta...@gmail.com>
AuthorDate: Wed Dec 18 12:46:49 2019 -0800
IMPALA-7117: Lower debug level for HDFS S3 connector back to INFO
This reverts the changes done in https://gerrit.cloudera.org/#/c/10596/
to set the HDFS S3 connector logs to DEBUG. DEBUG logs drastically increase
the size of the Impala logs during test runs and are no longer necessary
given recent stability improvements to the S3A code and the
Impala-S3Guard integration.
Change-Id: I4b8bf91fbf06e7609e3f92205e71557723163ecf
Reviewed-on: http://gerrit.cloudera.org:8080/14926
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
fe/src/test/resources/log4j.properties.template | 3 ---
1 file changed, 3 deletions(-)
diff --git a/fe/src/test/resources/log4j.properties.template b/fe/src/test/resources/log4j.properties.template
index 27ba520..31e2a48 100644
--- a/fe/src/test/resources/log4j.properties.template
+++ b/fe/src/test/resources/log4j.properties.template
@@ -26,6 +26,3 @@ log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
-
-# IMPALA-7008 / IMPALA-7070: Bump S3 log level to get more information.
-log4j.logger.org.apache.hadoop.fs.s3a=DEBUG
This is an automated email from the ASF dual-hosted git repository.
stakiar pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 32829b9594a90900a27c8117111a1b3499013e37
Author: Vihang Karajgaonkar <vi...@apache.org>
AuthorDate: Mon Dec 16 15:30:21 2019 -0800
IMPALA-9257: Last event id should be advanced if all events are skipped
Events processor implements a filtering method which skips certain
unnecessary events (eg. events on blacklisted dbs and tables).
However, if the received batch has all the events which are filtered
out, it fails to update its lastSyncedEventId. This causes unnecessary
logs being printed in catalog and the same event batch being fetched
repeatedly.
Testing:
Modified existing test to compare event id after events on blacklisted
dbs and tables.
Change-Id: I7f94c1a8e8c221f504262d5591cda8c3a25c0c32
Reviewed-on: http://gerrit.cloudera.org:8080/14916
Reviewed-by: Anurag Mantripragada <an...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
.../apache/impala/catalog/events/MetastoreEventsProcessor.java | 10 +++++++---
tests/custom_cluster/test_event_processing.py | 2 ++
2 files changed, 9 insertions(+), 3 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
index 22c1237..ac50562 100644
--- a/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
+++ b/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
@@ -584,19 +584,23 @@ public class MetastoreEventsProcessor implements ExternalEventsProcessor {
/**
* Process the given list of notification events. Useful for tests which provide a list
* of events
- *
- * @return the last Notification event which was processed.
*/
@VisibleForTesting
protected void processEvents(List<NotificationEvent> events)
throws MetastoreNotificationException {
NotificationEvent lastProcessedEvent = null;
+ // update the events received metric before returning
+ metrics_.getMeter(EVENTS_RECEIVED_METRIC).mark(events.size());
+ if (events.isEmpty()) return;
final Timer.Context context =
metrics_.getTimer(EVENTS_PROCESS_DURATION_METRIC).time();
- metrics_.getMeter(EVENTS_RECEIVED_METRIC).mark(events.size());
try {
List<MetastoreEvent> filteredEvents =
metastoreEventFactory_.getFilteredEvents(events);
+ if (filteredEvents.isEmpty()) {
+ lastSyncedEventId_.set(events.get(events.size() - 1).getEventId());
+ return;
+ }
for (MetastoreEvent event : filteredEvents) {
// synchronizing each event processing reduces the scope of the lock so the a
// potential reset() during event processing is not blocked for longer than
diff --git a/tests/custom_cluster/test_event_processing.py b/tests/custom_cluster/test_event_processing.py
index 0b7103b..7d67901 100644
--- a/tests/custom_cluster/test_event_processing.py
+++ b/tests/custom_cluster/test_event_processing.py
@@ -52,6 +52,7 @@ class TestEventProcessing(CustomClusterTestSuite):
event processor does not error out
"""
try:
+ event_id_before = EventProcessorUtils.get_last_synced_event_id()
self.run_stmt_in_hive("create database testBlackListedDb")
self.run_stmt_in_hive("create table testBlackListedDb.testtbl (id int)")
self.run_stmt_in_hive(
@@ -62,6 +63,7 @@ class TestEventProcessing(CustomClusterTestSuite):
# wait until all the events generated above are processed
EventProcessorUtils.wait_for_event_processing(self.hive_client)
assert EventProcessorUtils.get_event_processor_status() == "ACTIVE"
+ assert EventProcessorUtils.get_last_synced_event_id() > event_id_before
finally:
self.run_stmt_in_hive("drop database testBlackListedDb cascade")
self.run_stmt_in_hive("drop table functional_parquet.testBlackListedTbl")